10.10 01 Tomasz Kuchta

2020-03-01 58浏览

2.Tomasz Kuchta Senior Product Security Engineer Qualcomm Product Security Initiative (QPSI)Twitter:@Tomasz_KuchtaE-mail:tkuchta@qti.qualcomm.com ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
3.Agenda • Fuzzing is great! • Challenges of the mobile world and some of the solutions • Spreading theword:fuzzing across the organization • Beyondfuzzing:a fusion of techniques ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
5.Fuzzing • Fuzzing is a dynamic automated software testing technique • The target program is repeatedly tested with a modified input • The input can be random • It can be modified based on an algorithm (e.g. genetic operators in AFL) • Usually there is a feedback loop with an aim to increase code coverage • It can be used to automatically create test suites starting with a sample input • Generated test suites could be used with other tools – e.g. ASAN ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
6.Fuzzing types • Coverage-guided • The aim of the fuzzer is to increase code coverage • Coverage measured via compiler instrumentation, runtime instrumentation or HW counters • Mutation-based – e.g. AFL • Fuzzer applies a series of mutations to the input • For example, we can flip bits, mix generated inputs, randomize • Grammar-based – e.g. Peach • The user specifies an input grammar • For example, we can say that some parts of the input should not be modified by the fuzzer ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
7.Advantages:why do we like fuzzing • Scalability – ready to be used in the industry, works on real-world systems • Simplicity – the idea is conceptually simple to implement and deploy • No false positives – all bugs can be associated with a real input • Availability of proventools:AFL, syzkaller, libFuzzer, honggfuzz, etc. • Enables continuous testingefforts:best if done regularly and at scale • Results – it works! • AFL bug trophy page alone lists 161 different real-world software projects ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
9.In the modem world • Usual scenario for fuzzingprograms:• We have an executable • The executable can be fed with some input data (stdin/file/network) • The fuzzer starts with some inputs, mutates them and runs through the program • Our case – themodem:• A very complex system (millions LOC) with multiple tasks and state machines • Most (interesting) input comes OTA rather than from a file or stdin • As a result, some unique challenges arise ... ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
10.Challenge #1: hardware dependencies • Fuzzing can be done on target (using the original hardware): • That is the usual way for fuzzing applications • The whole software system runs normally on the target hardware •Example:generate and send WLAN messages to a mobile phone • It’s fine e.g. for native Linux apps, but it has limitations in the mobile context • Efficiency • Debugging • Expensive equipment necessary (like BTS) • Some protocols are not supported • Some of the equipment is difficult to configure and maintain ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
11.Solution #1: off-target fuzzing • We create an executable version of a system that can be run elsewhere • • • • Modify the code, make it compile and run on x86 and accept files as the inputResult:an executable program that can be fuzzed and easily debugged Much flexibility on the input injection pointsExample:• Extract WLAN message handling code and create a program out of it • Modify the program such that OTA messages are read from a file • Our off-target builds •Standard:testing framework allows entire protocol stack to run on x86 • The framework supports addition of test cases by the tech teams •Custom:files or features hand-picked ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
12.Challenge #2: choosing good entry points • Entrypoint:the place where the fuzzed messages are “injected” • In a command line program that could be stdin/file/network • In the modem that is OTA message • Injection pointmatters:• What do we want totest:the parser or the code after the parser? • How do we get to the right task and state machine ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
13.Solution #2: fuzz OTA message parsing points (input:file) Component Task 2 OTA Task 3 Parser A Parser B Task 1 ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
14.Challenge #3: stateful fuzzing (protocols) • State poses a challenge to fuzzing • Some code only reachable in certain state – e.g. protocol state machines • To fuzz some code we need to get to the state first • Before we get to “C” we need to be in “A” or “B” • Reproducibility of bugs is important A • If we find a crash – can we replay it with the same state? C B ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
15.Solution #3a:get the code to the right state first • That can be achieved through an off-target build • We modify the code such that it first sets up the right state • That is achieved via our off-target testing framework • Once the code is in the right state, the fuzzer-generated input is used ... A go_to_state_A(); C global_var = 1; go_to_state_B(); read_buffer_from_file(&buffer); B receive_message(buffer); ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
16.Solution #3b:treat one input as several messages • In a single fuzzer run • Take the input generated by the fuzzer • Use parts of the input as separate messages • That creates a message sequence which can get us further in the state machine Message 1 Message 2 Message 3 Message 4 Fuzzer-generated payload ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
17.Challenge #4: when do we stop? • It’s a tricky question and undecidable in a general case • What are the measurementcriteria:• Coverage? • Generic bugs? • Logic bugs? • Fuzzing time? • Can we guarantee the absence of bugs? ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
18.Solution #4: analysis feedback loop • We try to optimize the results via manual analysis feedback loop • Fuzz • Get the coverage • Analyze the report • Try to improve •Analysis:• Look for the parts of the code that were not covered • Should that code be covered? • Can that code be covered by using a fuzzer? ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
20.Thechallenge:how to split fuzzing work • What is the best split of work across security team and the tech teams? • Securityteam:limited resources as compared to those of tech teams • Techteams:tight schedules • The challenge is more on the organizational than on the technical level • Fuzzing could be centralized or distributed ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
21.The centralized model • Advantages of centralized fuzzingefforts:• Direct influence by the security team • Ownership of the fuzzing toolchain code/infrastructure S • However, there are also considerable drawbacks •Scalability:the security team is much smaller than the tech team •Priorities:given all the code we have, which targets should we fuzz first? •Expertise:the technical details best known by respective tech teams •Disclaimer:one size doesn’t fit all ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
22.Distributedmodel:the current working setup S Security team Tech teams • Started the process • Provides on-going support • Develop fuzzing infrastructure • Develop tests for their code bases • Small team responsible for driving the efforts • Transferring knowledge goes a long way • More and more tech team members become familiar with fuzzing know-how • As a security team we need experts in the tech teams • Scaleswell:most of on-going issues are solved by the tech teams •Disclaimer:one size doesn’t fit all ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
24.Multitude of available tools and techniques • • • • • • • • • • Static analysis, e.g. Klocwork, Coverity, FramaC Formal Verification, e.g. Coq, Dafny Bounded Model Checking, e.g. CBMC Fuzz testing, e.g. AFL, Honggfuzz, libFuzzer, syzcaller, PeachSanitizers:ASAN, MEMSAN, UBSAN, LSAN, TSAN VM-like, e.g. Valgrind, Intel PIN, DynamoRIO, BAP Symbolic execution, e.g. KLEE, Angr, Java PathFinder, PEX Invariant inference, e.g. CEGAR Taint tracking Program slicing ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
25.Toolslandscape:each tool is good/bad at something Memory errors Concurrency Scalability Ease of use Generates test cases Language specific Speed Platform specific No false positives ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
26.Toolslandscape:what would be great to have Ease of use No false positives Platform agnostic Generates test cases Speed Scalability Concurrency Memory errors Language agnostic ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
27.Could a fusion of various software analysis techniques capitalize on their advantages, alleviate their weaknesses and ultimately lead to “deeper” software testing? The research question ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
28.Merging software analysis techniques is not new • Sometimes it’s a single tool, sometimes multiple tools used together • Driller (symbex + fuzzing) [Stephens, et. al., NDSS 2016] • KleeFL (symbex + fuzzing) [Fietkau, et. al., Usenix Security 2017] • OSS-Fuzz (fuzzers + sanitizers) [Serebryany, et. al., 2016] • KATCH (symbex + static analysis) [Marinescu, et. al., ESEC/FSE 2013] • Frama-C (multiple analyzers) [Cuoq, et. al., SEFM 2012] • QSYM (concolic + fuzzing) [Yun, et. al., USENIX Security 2018] ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
30.Ourapproach:the pipeline • Easily configurable and extensible • N independent sequential steps • Manytools:step represents one tool Step 1 Step 2 … • One common set of inputs • All steps can access • All steps can contribute to • Pipeline runs in a loop Step N Controller Test inputs • Centralcontroller:• Runs steps • Controls test input set • Ideal deploymentscenario:large server ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
31.Step 1 Step 2 … Ourapproach:the pipeline Step N • Pipeline is parallelizable Step 1 Step 2 Step 1 … • All copies operate on a common global test input set Step N Step 2 … Step N Controller Test inputs ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
32.Thepipeline:potential advantages Centralized test caseset:• Tools can take advantage of test cases generated by others Decoupledarchitecture:• Inputs could be stored on different machine • Actual tools can also be run on different machine • The order and selection of tools is not hard-coded and can be easily changedController:• Provides guarantees like test input naming or synchronization • Could be used to define specialized search heuristics, e.g. • Pick first test cases going into certain branches • Feed tool A only with inputs generated by tool B ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
33.Pipeline we used klee afl asanFunctionality:• Symbolic execution • Fuzzing • Sanitizers • Dynamic analysis (Valgrind) • Code coverage (kcov) msan ubsan valgrind kcov Bugclasses:• Out of bound access • Use of uninitialized values • Undefined behaviors in C • Unsigned overflows • Memory leaks • NULL pointer dereference • Use after free • Division by zero • Assertion failures ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
34.Fuzzing is great, but also be aware of the limitations • Coverage is important but noteverything:covered != bug free • Being light-weight and scalable means that we lose reasoning power • But there are other techniques that can use generated inputs • Requires other tools for more thorough/deeper testing • Logical bugs – non-generic errors like unexpected output • Non-crashbugs:sanitizers, memory checkers ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
35.Reflections on the fusion • Combined functionality detects multiple bug classes • Example combinedstrengths:• Fuzzing scales well, but cannot reason about the program • Symbolic execution doesn’t scale that well, but has much more reasoning power • Sanitizers don’t generate test cases, but can verify them 01 int foo (int x, char* buffer) { 02 if (x > 5) { 03 printf(“Bar\n”); 04 return 1; 05 e.g. x == 10 } else if (x < 3) { 06 x++; 07 buffer[x] = ‘a‘; 08 } 09 return x; e.g. x == 0 10 } ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
36.Reflections on the fusion • Does fusing techniques help? Yes! • We can detect more classes of bugs with tools combined than alone • It‘s easier to use them if we have a framework (vs a number of separate scripts) • Framework also helps with parallelism • Somefindings:• It matters a lot how wetest:for us, testing each message type separately helped • Similarly-purposed tools might have different detectionrates:in our experiments Valgrind detected more occurrences than MEMSAN on the same inputs • Getting coverage is a must but tricky • How to measure? Instructions? Data? Backtrace? • If we covered 85% is that a lot or not? ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
38.Fuzzing in the mobile world • Fuzzing is great for theindustry:it works out of the box, scales and delivers • Mobile ecosystem has it’s own uniquechallenges:• • • • • HW dependencies Selecting right fuzzing input points in very complex systems Stateful fuzzing How to decide we fuzzed enough? What is the best split of fuzzing work across security team and tech teams? • Ultimately, every testing technique is limited, but we can try to use them together ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
39.References •https://www.sec-consult.com/wpcontent/uploads/files/vulnlab/the_art_of_fuzzing_slides.pdf•https://resources.infosecinstitute.com/fuzzing-mutation-vsgeneration/#gref •https://en.wikipedia.org/wiki/Fuzzing•AFL:http://lcamtuf.coredump.cx/afl/•honggfuzz:http://honggfuzz.com/•syzkaller:https://github.com/google/syzkaller•libFuzzer:https://llvm.org/docs/LibFuzzer.html•KLEE:http://klee.github.io/•Peach:https://www.peach.tech/©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
40.Qualcomm Vulnerability Rewards Program • We are looking for researchers • More details, including targets and rewards, can be foundat:https://hackerone.com/qualcomm• The program is invite-only • If you are interested, please contact us at product-security@qualcomm.com • Please provide references to your prior work, e.g. CVEs, Android bug IDs ©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
41.Thank youhttps://www.qualcomm.com/product-securityQualcomm Vulnerability Rewards Program is looking for researchershttps://hackerone.com/qualcomm©2018 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
42.
43.
44.
45.