Accelerating soil insights with cost-effective PDF data extraction at Stantec

Massive data volume: Stantec manages the Soil Risk Map (SRM) to rapidly assess excavation risks. This map utilizes an archive of 250,000 PDF files containing soil analysis certificates—a collection that grows by 10,000 to 20,000 documents each month.
Manual processing is unfeasible, given that some documents run to thousands of pages.

Slow and limited PoC: The existing Proof of Concept (PoC) successfully processed only 12% of the documents and operated extremely slowly. he largest documents took several minutes to process due to slow Python packages such as PdfPlumber and Camelot-python.

Strict quality and cloud requirements: Because overlooking a soil measurement (such as for asbestos) can have major operational consequences, very strict quality standards apply.
Additionally, cloud and data access were restricted due to an ongoing migration to Microsoft Azure, meaning testing during development could only be performed on approximately 1% of the total document corpus.

Faster PDF ingestion via XPDF (XPyDF): Slow Python libraries were replaced by XPDF (written in C++), which supports pdftotext functionality while preserving character positions for table parsing. BigData Republic developed the open-source Python wrapper XPyDF for this purpose.

Strict error-handling framework: A custom error-handling framework with detailed error messages was implemented, ensuring the pipeline stops early and in a controlled manner whenever an unexpected scenario occurs.

Performance analysis using Sankey diagrams: A visualization tool using Plotly Sankey diagrams was built to monitor performance changes in a data-driven way.

Efficient Azure cloud architecture: The solution was deployed on Microsoft Azure using a Function App, a Container Instance, and three storage containers (Input, Data-batch, and Results).