Extend integration test to test sequence of FUSE operation

Original FUSE integration test has limited capabilities. To test more
situations, the new integration test framework introduces a protocol
to communicate between testing thread and the FUSE server. In summary,
this change includes:

1. Remove CompareResult() and break SetExpected() into
   SetServerResponse() and GetServerActualRequest(). We no longer set
   up an expected request because we want to retrieve the actual FUSE
   request made to the FUSE server and check in the testing thread.

2. Declare a serial buffer data structure to save the received requests
   and expected responses sequentially. The data structure contains a
   cursor to indicate the progress of accessing. This change makes
   sequential SetServerResponse() and GetServerActualRequest() possible.

3. Replace 2 single directional pipes with 1 bi-directional socketpair.
   A protocol which starts with FuseTestCmd is used between the testing
   thread and the FUSE server to provide various functionality.

Fixes #3405

1 files changed, 118 insertions, 47 deletions

diff --git a/test/fuse/README.md b/test/fuse/README.md
index 734c3a4e3..c5909a166 100644
--- a/test/fuse/README.md
+++ b/test/fuse/README.md
@@ -1,55 +1,90 @@
 # gVisor FUSE Test Suite
 
-This is an integration test suite for fuse(4) filesystem. It runs under both
-gVisor and Linux, and ensures compatibility between the two. This test suite is
-based on system calls test.
+This is an integration test suite for fuse(4) filesystem. It runs under gVisor
+sandbox container with VFS2 and FUSE function enabled.
 
-This document describes the framework of fuse integration test and the
-guidelines that should be followed when adding new fuse tests.
+This document describes the framework of FUSE integration test, how to use it,
+and the guidelines that should be followed when adding new testing features.
 
 ## Integration Test Framework
 
-Please refer to the figure below. `>` is entering the function, `<` is leaving
-the function, and `=` indicates sequentially entering and leaving.
+By inheriting the `FuseTest` class defined in `linux/fuse_base.h`, every test
+fixture can run in an environment with `mount_point_` mounted by a fake FUSE
+server. It creates a `socketpair(2)` to send and receive control commands and
+data between the client and the server. Because the FUSE server runs in the
+background thread, gTest cannot catch its assertion failure immediately. Thus,
+`TearDown()` function sends command to the FUSE server to check if all gTest
+assertion in the server are successful and all requests and preset responses
+are consumed.
+
+## Communication Diagram
+
+Diagram below describes how a testing thread communicates with the FUSE server
+to achieve integration test.
+
+For the following diagram, `>` means entering the function, `<` is leaving the
+function, and `=` indicates sequentially entering and leaving. Not necessarily
+follow exactly the below diagram due to the nature of a multi-threaded system,
+however, it is still helpful to know when the client waits for the server to
+complete a command and when the server awaits the next instruction.
 
 ```
- |  Client (Test Main Process)         |  Server (FUSE Daemon)
+ |  Client (Testing Thread)            |  Server (FUSE Server Thread)
  |                                     |
  |  >TEST_F()                          |
  |    >SetUp()                         |
  |      =MountFuse()                   |
  |      >SetUpFuseServer()             |
- |        [create communication pipes] |
- |        =fork()                      |        =fork()
- |        >WaitCompleted()             |
- |          [wait for MarkDone()]      |
- |                                     |        =ConsumeFuseInit()
- |                                     |        =MarkDone()
- |        <WaitCompleted()             |
+ |        [create communication socket]|
+ |        =fork()                      |      =fork()
+ |        [wait server complete]       |
+ |                                     |      =ServerConsumeFuseInit()
+ |                                     |      =ServerCompleteWith()
  |      <SetUpFuseServer()             |
  |    <SetUp()                         |
- |    >SetExpected()                   |
- |      [construct expected reaction]  |
- |                                     |        >FuseLoop()
- |                                     |          >ReceiveExpected()
- |                                     |            [wait data from pipe]
- |      [write data to pipe]           |
- |      [wait for MarkDone()]          |
+ |    [testing main]                   |
+ |                                     |      >ServerFuseLoop()
+ |                                     |        [poll on socket and fd]
+ |    >SetServerResponse()             |
+ |      [write data to socket]         |
+ |      [wait server complete]         |
+ |                                     |        [socket event occurs]
+ |                                     |        >ServerHandleCommand()
+ |                                     |          >ServerReceiveResponse()
+ |                                     |            [read data from socket]
  |                                     |            [save data to memory]
- |                                     |            =MarkDone()
- |    <SetExpected()                   |
- |                                     |          <ReceiveExpected()
- |                                     |          >read()
- |                                     |            [wait for fs operation]
+ |                                     |          <ServerReceiveResponse()
+ |                                     |          =ServerCompleteWith()
+ |    <SetServerResponse()             |
+ |                                     |        <ServerHandleCommand()
  |    >[Do fs operation]               |
  |      [wait for fs response]         |
- |                                     |          <read()
- |                                     |          =CompareRequest()
- |                                     |          =write() [write fs response]
+ |                                     |        [fd event occurs]
+ |                                     |        >ServerProcessFuseRequest()
+ |                                     |          =[read fs request]
+ |                                     |          =[save fs request to memory]
+ |                                     |          =[write fs response]
  |    <[Do fs operation]               |
+ |                                     |        <ServerProcessFuseRequest()
+ |                                     |
  |    =[Test fs operation result]      |
- |    =[wait for MarkDone()]           |
- |                                     |          =MarkDone()
+ |                                     |
+ |    >GetServerActualRequest()        |
+ |      [write data to socket]         |
+ |      [wait data from server]        |
+ |                                     |        [socket event occurs]
+ |                                     |        >ServerHandleCommand()
+ |                                     |          >ServerSendReceivedRequest()
+ |                                     |            [write data to socket]
+ |      [read data from socket]        |
+ |      [wait server complete]         |
+ |                                     |          <ServerSendReceivedRequest()
+ |                                     |          =ServerCompleteWith()
+ |    <GetServerActualRequest()        |
+ |                                     |        <ServerHandleCommand()
+ |                                     |
+ |    =[Test actual request]           |
+ |                                     |
  |    >TearDown()                      |
  |      =UnmountFuse()                 |
  |    <TearDown()                      |
@@ -58,8 +93,8 @@ the function, and `=` indicates sequentially entering and leaving.
 
 ## Running the tests
 
-Based on syscall tests, fuse tests can run in different environments. To enable
-fuse testing environment, the test targets should be appended with `_fuse`.
+Based on syscall tests, FUSE tests generate targets only with vfs2 and fuse
+enabled. The corresponding targets end in `_fuse`.
 
 For example, to run fuse test in `stat_test.cc`:
 
@@ -75,19 +110,17 @@ $ bazel test --test_tag_filters=fuse //test/fuse/...
 
 ## Writing a new FUSE test
 
-1.  Add test targets in `BUILD` and `linux/BUILD`.
-2.  Inherit your test from `FuseTest` base class. It allows you to:
-    -   Run a fake FUSE server in background during each test setup.
-    -   Create pipes for communication and provide utility functions.
-    -   Stop FUSE server after test completes.
-3.  Customize your comparison function for request assessment in FUSE server.
-4.  Add the mapping of the size of structs if you are working on new FUSE
-    opcode.
-    -   Please update `FuseTest::GetPayloadSize()` for each new FUSE opcode.
-5.  Build the expected request-response pair of your FUSE operation.
-6.  Call `SetExpected()` function to inject the expected reaction.
-7.  Check the response and/or errors.
-8.  Finally call `WaitCompleted()` to ensure the FUSE server acts correctly.
+1. Add test targets in `BUILD` and `linux/BUILD`.
+2. Inherit your test from `FuseTest` base class. It allows you to:
+  - Fork a fake FUSE server in background during each test setup.
+  - Create a pair of sockets for communication and provide utility functions.
+  - Stop FUSE server and check if error occurs in it after test completes.
+3. Build the expected opcode-response pairs of your FUSE operation.
+4. Call `SetServerResponse()` to preset the next expected opcode and response.
+5. Do real filesystem operations (FUSE is mounted at `mount_point_`).
+6. Check FUSE response and/or errors.
+7. Retrieve FUSE request by `GetServerActualRequest()`.
+8. Check if the request is as expected.
 
 A few customized matchers used in syscalls test are encouraged to test the
 outcome of filesystem operations. Such as:
@@ -101,3 +134,41 @@ SyscallFailsWithErrno(...)
 
 Please refer to [test/syscalls/README.md](../syscalls/README.md) for further
 details.
+
+## Writing a new FuseTestCmd
+
+A `FuseTestCmd` is a control protocol used in the communication between the
+testing thread and the FUSE server. Such commands are sent from the testing
+thread to the FUSE server to set up, control, or inspect the behavior of the
+FUSE server in response to a sequence of FUSE requests.
+
+The lifecycle of a command contains following steps:
+
+1. The testing thread sends a `FuseTestCmd` via socket and waits for completion.
+2. The FUSE server receives the command and does corresponding action.
+3. (Optional) The testing thread reads data from socket.
+4. The FUSE server sends a success indicator via socket after processing.
+5. The testing thread gets the success signal and continues testing.
+
+The success indicator, i.e. `WaitServerComplete()`, is crucial at the end of
+each `FuseTestCmd` sent from the testing thread. Because we don't want to begin
+filesystem operation if the requests have not been completely set up. Also, to
+test FUSE interactions in a sequential manner, concurrent requests are not
+supported now.
+
+To add a new `FuseTestCmd`, one must comply with following format:
+
+1. Add a new `FuseTestCmd` enum class item defined in `linux/fuse_base.h`
+2. Add a `SetServerXXX()` or `GetServerXXX()` public function in `FuseTest`.
+   This is how the testing thread will call to send control message. Define how
+   many bytes you want to send along with the command and what you will expect
+   to receive. Finally it should block and wait for a success indicator from
+   the FUSE server.
+3. Add a `ServerReceiveXXX()` or `ServerSendXXX()` private function in
+   `FuseTest`. It is mandatory to set it private since only the FUSE server
+   (forked from `FuseTest` base class) can call it. This is the handler of a
+   specific `FuseTestCmd` and the format of the data should be consistent with
+   what client expects in the previous step.
+4. Add a case in the switch condition of `ServerHandleCommand()` to route the
+   command to the server handler described in the previous step.
+