轻量级虚拟化技术草稿

Support Tech

ST.1 virtiofs

ST.1.1 fuse framework

引用wiki中关于fuse的定义：

Filesystem in Userspace (FUSE) is a software interface for Unix and Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code. This is achieved by running file system code in user space while the FUSE module provides only a bridge to the actual kernel interfaces.

其代码框架如下： 

主要分为三部分：

•  内核Fuse Filsystem Client，对接Linux Kernel的VFS
• Fuse协议，其中包括了op code及其参数格式，参考GitHub - libfuse/libfuse: The reference implementation of the Linux FUSE (Filesystem in Userspace) interfaceThe reference implementation of the Linux FUSE (Filesystem in Userspace) interface - GitHub - libfuse/libfuse: The reference implementation of the Linux FUSE (Filesystem in Userspace) interfacehttps://github.com/libfuse/libfuse
•  Fuse传输层，目前包括两种，本地char设备和virtio

参考代码：

fuse_do_readpage()
---...loff_t pos = page_offset(page);struct fuse_page_desc desc = { .length = PAGE_SIZE };struct fuse_io_args ia = {.ap.args.page_zeroing = true,.ap.args.out_pages = true,.ap.num_pages = 1,.ap.pages = &page,.ap.descs = &desc,};...fuse_wait_on_page_writeback(inode, page->index);...fuse_read_args_fill(&ia, file, pos, desc.length, FUSE_READ);res = fuse_simple_request(fm, &ia.ap.args);...SetPageUptodate(page);
---fuse_lookup_name()
---fuse_lookup_init(fm->fc, &args, nodeid, name, outarg);---args->opcode = FUSE_LOOKUP;args->nodeid = nodeid;args->in_numargs = 1;args->in_args[0].size = name->len + 1;args->in_args[0].value = name->name;args->out_numargs = 1;args->out_args[0].size = sizeof(struct fuse_entry_out);args->out_args[0].value = outarg;// fuse_entry_out.attr includes all of inode attributes, such as ino/size/blocks/atime/mtime/ctime/nlink/mode/uid/gid ...---err = fuse_simple_request(fm, &args);...*inode = fuse_iget(sb, outarg->nodeid, outarg->generation,&outarg->attr, entry_attr_timeout(outarg),attr_version);
---fuse_simple_request()
---if (args->force) {atomic_inc(&fc->num_waiting);req = fuse_request_alloc(fm, GFP_KERNEL | __GFP_NOFAIL);...__set_bit(FR_WAITING, &req->flags);__set_bit(FR_FORCE, &req->flags);} else {req = fuse_get_req(fm, false);...}...__fuse_request_send(req);...fuse_put_request(req);
---__fuse_request_send()-> spin_lock(&fiq->lock);-> queue_request_and_unlock()---list_add_tail(&req->list, &fiq->pending);fiq->ops->wake_pending_and_unlock(fiq);----> request_wait_answer()---if (!fc->no_interrupt) {/* Any signal may interrupt this */err = wait_event_interruptible(req->waitq,test_bit(FR_FINISHED, &req->flags));...}if (!test_bit(FR_FORCE, &req->flags)) {/* Only fatal signals may interrupt this */err = wait_event_killable(req->waitq,test_bit(FR_FINISHED, &req->flags));...}/** Either request is already in userspace, or it was forced.* Wait it out.*/wait_event(req->waitq, test_bit(FR_FINISHED, &req->flags));---

以上代码中列举了两个常见的文件系统操作，readpage和lookup，它们都是同步的，所以，需要request_wait_answer()。

write page的处理代码如下：

fuse_writepage_locked()
---tmp_page = alloc_page(GFP_NOFS | __GFP_HIGHMEM);...fuse_write_args_fill(&wpa->ia, wpa->ia.ff, page_offset(page), 0);copy_highpage(tmp_page, page);wpa->ia.write.in.write_flags |= FUSE_WRITE_CACHE;wpa->next = NULL;ap->args.in_pages = true;ap->num_pages = 1;ap->pages[0] = tmp_page;ap->descs[0].offset = 0;ap->descs[0].length = PAGE_SIZE;ap->args.end = fuse_writepage_end;wpa->inode = inode;inc_wb_stat(&inode_to_bdi(inode)->wb, WB_WRITEBACK);inc_node_page_state(tmp_page, NR_WRITEBACK_TEMP);spin_lock(&fi->lock);tree_insert(&fi->writepages, wpa);list_add_tail(&wpa->queue_entry, &fi->queued_writes);fuse_flush_writepages(inode);spin_unlock(&fi->lock);end_page_writeback(page);
---fuse_writepages()
---err = write_cache_pages(mapping, wbc, fuse_writepages_fill, &data);if (data.wpa) {WARN_ON(!data.wpa->ia.ap.num_pages);fuse_writepages_send(&data);}
---fuse_writepages_send()
---spin_lock(&fi->lock);list_add_tail(&wpa->queue_entry, &fi->queued_writes);fuse_flush_writepages(inode);spin_unlock(&fi->lock);for (i = 0; i < num_pages; i++)end_page_writeback(data->orig_pages[i]);
---fuse_flush_writepages()-> fuse_send_writepage()-> fuse_simple_background()-> fuse_request_queue_background()---if (likely(fc->connected)) {fc->num_background++;if (fc->num_background == fc->max_background)fc->blocked = 1;if (fc->num_background == fc->congestion_threshold && fm->sb) {set_bdi_congested(fm->sb->s_bdi, BLK_RW_SYNC);set_bdi_congested(fm->sb->s_bdi, BLK_RW_ASYNC);}list_add_tail(&req->list, &fc->bg_queue);flush_bg_queue(fc);queued = true;}---fuse_send_writepage()
---fi->writectr++;
---fuse_writepage_end()
---fi->writectr--;fuse_writepage_finish(fm, wpa);---for (i = 0; i < ap->num_pages; i++) {dec_wb_stat(&bdi->wb, WB_WRITEBACK);dec_node_page_state(ap->pages[i], NR_WRITEBACK_TEMP);wb_writeout_inc(&bdi->wb);}wake_up(&fi->page_waitq);---
---fuse_fsync()-> file_write_and_wait_range()-> fuse_sync_writes(inode)-> fuse_set_nowrite()---spin_lock(&fi->lock);BUG_ON(fi->writectr < 0);fi->writectr += FUSE_NOWRITE;spin_unlock(&fi->lock);wait_event(fi->page_waitq, fi->writectr == FUSE_NOWRITE);---
---

write page中最大的不同在于：fuse申请了一个tmp_page，然后使用这个tmp_page去承载数据，并发给对端；发出去之后，立刻就调用了end_page_writeback()，此时page cache中的数据还没有真正落盘；数据落盘的语义最终是通过fsync保证的。这样就避免了用户态fuse daemon故障导致系统dirty pages无人处理。

对与这个设计，fuse的commit的comment中做了解释：

    Fuse page writeback design--------------------------fuse_writepage() allocates a new temporary page with GFP_NOFS|__GFP_HIGHMEM.It copies the contents of the original page, and queues a WRITE request to theuserspace filesystem using this temp page.The writeback is finished instantly from the MM's point of view: the page isremoved from the radix trees, and the PageDirty and PageWriteback flags arecleared.For the duration of the actual write, the NR_WRITEBACK_TEMP counter isincremented.  The per-bdi writeback count is not decremented until the actualwrite completes.On dirtying the page, fuse waits for a previous write to finish beforeproceeding.  This makes sure, there can only be one temporary page used at atime for one cached page.This approach is wasteful in both memory and CPU bandwidth, so why is this^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^complication needed?The basic problem is that there can be no guarantee about the time in which^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^the userspace filesystem will complete a write.  It may be buggy or even^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^malicious, and fail to complete WRITE requests.  We don't want unrelated partsof the system to grind to a halt in such cases.Also a filesystem may need additional resources (particularly memory) tocomplete a WRITE request.  There's a great danger of a deadlock if thatallocation may wait for the writepage to finish.