How Fast Can You Parse a File with 1 Billion Rows of Weather Data Using Java?
This video is also available in the GOTO Play video app! Download it to enjoy offline access to our conference videos while on the move.
or
Last January a challenge was posted online by Gunnar Morling: How fast can you parse a file with one billion rows of weather data using Java?
Little did I know this deceivingly simple question would lead me down a path that taught me all about: parallelism, memory mapped files, SWAR techniques (SIMD as a register), bit twiddling, branchless code, mechanical sympathy, Graal native compilation and finally... I even turned to the dark side: using sun.misc.Unsafe.
Join me in this deep dive where I'll explain all the code changes and tricks that took me from the reference implementation which processes the billion records in >4 minutes, to processing everything in under two seconds.
Who knew Java could be this fast?
-
A Short Summary of the Last Decades of Data ManagementHannes MühleisenTuesday Jun 11 @ 13:25
-
Lessons From The Pit LaneMarc PriestleyWednesday Jun 12 @ 09:10
-
X Marks the Spot: Navigating Possible Futures with Wardley MapsSimon WardleyWednesday Jun 12 @ 16:30
-
Is It Time To Version Observability? (Signs Point To Yes)Charity MajorsTuesday Jun 11 @ 09:10
-
How a Passion for Oceans Can Utilize Synergies of TechnologySigne SimonsenWednesday Jun 12 @ 13:25
-
There’s no AI in human: Navigating The Intersection of Technology and HumanityImran RashidTuesday Jun 11 @ 17:10