How fast can you parse a file with one billion rows of weather data using Java?
Last January a challenge was posted online by Gunnar Morling: How fast can you parse a file with one billion rows of weather data using Java?
Little did I know this deceivingly simple question would lead me down a path that taught me all about: parallelism, memory mapped files, SWAR techniques (SIMD as a register), bit twiddling, branchless code, mechanical sympathy, Graal native compilation and finally... I even turned to the dark side: using sun.misc.Unsafe.
Join me in this deep dive where I'll explain all the code changes and tricks that took me from the reference implementation which processes the billion records in >4 minutes, to processing everything in under two seconds.
Who knew Java could be this fast?
-
A Short Summary of the Last Decades of Data ManagementHannes MühleisenTuesday Jun 11, 13:25
-
Lessons From The Pitlane with Marc PriestleyMarc PriestleyWednesday Jun 12, 09:10
-
X Marks the Spot: Navigating Possible Futures with Wardley MapsSimon WardleyWednesday Jun 12, 16:30
-
Compliance & Regulatory Standards Are NOT Incompatible With Modern Development Best PracticesCharity MajorsTuesday Jun 11, 09:10
-
How a passion for oceans can utilize synergies of technology.Signe SimonsenWednesday Jun 12, 13:25
-
There’s no AI in human: Navigating The Intersection of Technology and HumanityImran RashidTuesday Jun 11, 17:10