5

I am trying to extract OSRM data but having troubles. In a previous question (lua error when trying to extract osrm data), I was encountering an error. So I am not trying to do the same thing except with Docker. I am using a newer Mac with a Silicon chip. I first installed Docker (https://docs.docker.com/desktop/install/mac-install/; and then in the Terminal docker pull osrm/osrm-backend). I download the North America data from https://download.geofabrik.de/north-america.html. I have allocated 50GB of memory to the instance as well as several cores. If, in Terminal, I run docker run -t -v /Users/[name]/osrm-backend/docker:/data osrm/osrm-backend osrm-extract -p /opt/car.lua /data/north-america-latest.osm.pbf, I get:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /opt/car.lua
[info] Input file: north-america-latest.osm.pbf
[info] Profile: car.lua
[info] Threads: 5
[info] Parsing in progress..
[info] input file generated by osmium/1.14.0
[info] timestamp: 2023-01-05T21:21:39Z
[info] Using profile api version 4
[info] Found 3 turn restriction tags:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parse relations ...
TBB Warning: Exact exception propagation is requested by application but the linked library is built without support for it
terminate called after throwing an instance of 'tbb::captured_exception'
  what():  PBF error: truncated data (EOF encountered)
qemu: uncaught target signal 6 (Aborted) - core dumped

The process might still be running (meaning that I can't execute any new commands), but hasn't apparently done anything for quite a while.

What do these warnings mean, and how do I fix them?

Note also that if I run another file, it works. docker run -t -v /Users/[name]/osrm-backend/docker:/data osrm/osrm-backend osrm-extract -p /opt/car.lua /data/us-virgin-islands-latest.osm.pbf

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /opt/car.lua
[info] Input file: us-virgin-islands-latest.osm.pbf
[info] Profile: car.lua
[info] Threads: 5
[info] Parsing in progress..
[info] input file generated by osmium/1.14.0
[info] timestamp: 2023-01-05T21:21:39Z
[info] Using profile api version 4
[info] Found 3 turn restriction tags:
[info]   motorcar
[info]   motor_vehicle
[info]   vehicle
[info] Parse relations ...
[info] Parse ways and nodes ...
[info] Using profile api version 4
[info] Using profile api version 4
[info] Using profile api version 4
[info] Using profile api version 4
[info] Parsing finished after 4.90155 seconds
[info] Raw input contains 1316532 nodes, 72554 ways, and 97 relations, 30 restrictions
[info] Collecting way information on 30 restrictions...ok, after 0.000596s
[info] Collecting way information on 0 maneuver overrides...ok, after 0.000106s
[info] Sorting used nodes        ... ok, after 0.004332s
[info] Erasing duplicate nodes   ... ok, after 0.000402s
[info] Sorting all nodes         ... ok, after 0.002219s
[info] Building node id map      ... ok, after 0.00384s
[info] Confirming/Writing used nodes     ... ok, after 0.111959s
[info] Writing barrier nodes     ... ok, after 0s
[info] Writing traffic light nodes     ... ok, after 0s
[info] Processed 73181 nodes
[info] Sorting edges by start    ... ok, after 0.006168s
[info] Setting start coords      ... ok, after 0.010829s
[info] Sorting edges by target   ... ok, after 0.006618s
[info] Computing edge weights    ... ok, after 0.111005s
[info] Sorting edges by renumbered start ... ok, after 0.01132s
[info] Writing used edges       ... ok, after 0.013881s -- Processed 75963 edges
[info] Writing way meta-data     ... ok, after 0.001686s -- Metadata contains << 9487 entries.
[info] Collecting node information on 0 maneuver overrides...ok, after 8e-06s
[info] Collecting node information on 30 restrictions...ok, after 0.000959s
[info] writing street name index ... ok, after 0.004272s
[info] extraction finished after 5.21655s
[info] Generating edge-expanded graph representation
[info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
[info] Node compression ratio: 0.19777
[info] Edge compression ratio: 0.220645
[info]  graph compression removed 8425 annotations of 9487 in 0.014951 seconds
[info] Find segregated edges in node-based graph ...
[info] ok, after 0.426069s
[info] Segregated edges count = 57
[info] Writing nodes for nodes-based and edges-based graphs ...
[info] Geometry successfully removed:
  compressed edges: 33300
  compressed geometries: 150658
  longest chain length: 151
  cmpr ratio: 0.22103
  avg chain length: 4.52426
[info] Removing invalid restrictions...removed 0 invalid restrictions, after 0.000302s
[info] Constructing restriction graph on 30 restrictions...ok, after 0.001196s
[info] Generating edge expanded nodes ... 
[info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
[info] Expanding via-way turn restrictions ... 
[info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80%  100%
[info] Generated 32790 nodes (5 of which are duplicates)  and 75329 segments in edge-expanded graph
[info] Generating edge-expanded edges 
[info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% .
[info] Sorting and writing 0 maneuver overrides...
[info] done.
[info] Renumbering turns
[info] Writing 0 conditional turn penalties...
[info] Generated 75329 edge based node segments
[info] Node-based graph contains 32735 edges
[info] Edge-expanded graph ...
[info]   contains 61286 edges
[info] Timing statistics for edge-expanded graph:
[info] Renumbering edges: 0.001581s
[info] Generating nodes: 0.01637s
[info] Generating edges: 0.7993s
[info] Generating guidance turns 
[info] . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% .
[info] done.
[info] Created 26 entry classes and 1550 Bearing Classes
[info] Handled: 0 of 13 lanes: 0 %.
[info] Assigned 68762 turn instruction types:
[info]   new name: 3076 (4.47%)
[info]   continue: 5507 (8.01%)
[info]   turn: 27959 (40.66%)
[info]   merge: 21 (0.03%)
[info]   on ramp: 5 (0.01%)
[info]   off ramp: 8 (0.01%)
[info]   fork: 832 (1.21%)
[info]   end of road: 10601 (15.42%)
[info]   notification: 4 (0.01%)
[info]   enter roundabout: 6 (0.01%)
[info]   enter roundabout turn: 11 (0.02%)
[info]   (noturn): 4261 (6.20%)
[info]   (suppressed): 16403 (23.85%)
[info]   exit roundabout: 8 (0.01%)
[info]   exit roundabout turn: 11 (0.02%)
[info]   (stay on roundabout): 19 (0.03%)
[info]   (sliproad): 30 (0.04%)
[info] Assigned 68762 turn instruction modifiers:
[info]   uturn: 5105 (7.42%)
[info]   sharp right: 2730 (3.97%)
[info]   right: 16456 (23.93%)
[info]   slight right: 2962 (4.31%)
[info]   straight: 19307 (28.08%)
[info]   slight left: 2989 (4.35%)
[info]   left: 16526 (24.03%)
[info]   sharp left: 2687 (3.91%)
[info] Guidance turn annotations took 1.18582s
[info] Writing Intersection Classification Data
[info] ok, after 0.00886s
[info] Writing Turns and Lane Data...
[info] ok, after 0.01241s
[info] Saving edge-based node weights to file.
[info] Done writing. (0.005628)
[info] Computing strictly connected components ...
[info] Found 64 SCC (2 large, 62 small)
[info] SCC run took: 0.00466692s
[info] Building r-tree ...
[info] Constructing r-tree of 75329 segments build on-top of 73181 coordinates
[info] finished r-tree construction in 0.149264 seconds
[info] Writing edge-based-graph edges       ... 
[info] ok, after 0.121746s
[info] Processed 61286 edges
[info] Expansion: 26516 nodes/sec and 11881 edges/sec
[info] To prepare the data for routing, run: ./osrm-contract "/data/us-virgin-islands-latest.osrm"
[info] RAM: peak bytes used: 183009280

2 Answers 2

1

It might be a memory issue, try to allocate more. As somebody mentioned in this issue https://github.com/Project-OSRM/osrm-backend/issues/5614#issuecomment-560504788:

The only real reference point I have is that it takes about 100GB of RAM to process North America.

Sign up to request clarification or add additional context in comments.

Comments

1

Try downloading the file /data/north-america-latest.osm.pbf again or verify if it downloaded successfully.

It could be corrupt due to a broken download.

The error you are facing seems to match here - https://github.com/peter-evans/osrm-backend-k8s/issues/48

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.