class="tags" href="/tags/RDMA.html" title=RDMA>RDMA_0">class="tags" href="/tags/RDMA.html" title=RDMA>RDMA编程实践

本文描述了class="tags" href="/tags/RDMA.html" title=RDMA>RDMA编程过程中的SEND-RECEIVE双边原语的代码实现。包含多个版本࿰c;1、client向server发送消息࿰c;server回复client收到消息(ACK)࿰c;然后两边断开连接。2、server端循环等待客户端建立连接࿰c;client发送一次消息后࿰c;双方断开连接。3、server端循环等待客户端建立连接࿰c;一旦建立࿰c;client端可以一直向server端发送消息࿰c;直到发送消息为disconnect࿰c;server和client断开链接࿰c;但是server此时仍然可以等待别的client发送消息。
代码基于代码基于class="tags" href="/tags/SEND-RECEIVE.html" title=send-receive>send-receive样例实现。关于代码注释࿰c;可以参考代码解释：
Makefile文件、会编译当前目录下的所有.c文件：

<code>.PHONY: all clean

CC := gcc
CFLAGS := -Wall -g
LDLIBS := -lrdmacm -libverbs -lpthread -g

SRCS := $(wildcard *.c)
APPS := $(SRCS:.c=)

all: $(APPS)

%: %.c
	$(CC) $(CFLAGS) $< -o $@ $(LDLIBS)

clean:
	rm -f $(APPS)
code>

version1 客户端-服务端消息一次传递

在这个阶段࿰c;我们希望能实现下面这样一个场景。client与server端相连接࿰c;client端能够发送一条消息给server࿰c;server收到该条消息之后恢复一条消息给client端表示我已经确认收到。之后两者断开连接。

代码：

<code class="prism language-c">class="token comment">// client1.c
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdio.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdlib.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><string.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><netdb.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><errno.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><getopt.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_cma.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_verbs.h>

class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*server class="token operator">= class="token string">"10.10.10.1"class="token punctuation">;
class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*port class="token operator">= class="token string">"7471"class="token punctuation">;

class="token keyword">static class="token keyword">struct class="token class-name">rdma_cm_id class="token operator">*idclass="token punctuation">;
class="token keyword">static class="token keyword">struct class="token class-name">ibv_mr class="token operator">*mrclass="token punctuation">, class="token operator">*send_mrclass="token punctuation">;
class="token keyword">static class="token keyword">int send_flagsclass="token punctuation">;
class="token keyword">static class="token class-name">uint8_t send_msgclass="token punctuation">[class="token number">16class="token punctuation">]class="token punctuation">;
class="token keyword">static class="token class-name">uint8_t recv_msgclass="token punctuation">[class="token number">16class="token punctuation">]class="token punctuation">;

class="token keyword">static class="token keyword">int class="token function">runclass="token punctuation">(class="token keyword">voidclass="token punctuation">)
class="token punctuation">{
	class="token keyword">struct class="token class-name">rdma_addrinfo hintsclass="token punctuation">, class="token operator">*resclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_qp_init_attr attrclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_wc wcclass="token punctuation">;
	class="token keyword">int retclass="token punctuation">;

	class="token function">memsetclass="token punctuation">(class="token operator">&hintsclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof hintsclass="token punctuation">)class="token punctuation">;
	hintsclass="token punctuation">.ai_port_space class="token operator">= class="tags" href="/tags/RDMA.html" title=RDMA>RDMA_PS_TCPclass="token punctuation">;
	ret class="token operator">= class="token function">rdma_getaddrinfoclass="token punctuation">(serverclass="token punctuation">, portclass="token punctuation">, class="token operator">&hintsclass="token punctuation">, class="token operator">&resclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">printfclass="token punctuation">(class="token string">"rdma_getaddrinfo: %s\n"class="token punctuation">, class="token function">gai_strerrorclass="token punctuation">(retclass="token punctuation">)class="token punctuation">)class="token punctuation">;
		class="token keyword">goto outclass="token punctuation">;
	class="token punctuation">}

	class="token function">memsetclass="token punctuation">(class="token operator">&attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof attrclass="token punctuation">)class="token punctuation">;
	attrclass="token punctuation">.capclass="token punctuation">.max_send_wr class="token operator">= attrclass="token punctuation">.capclass="token punctuation">.max_recv_wr class="token operator">= class="token number">1class="token punctuation">;
	attrclass="token punctuation">.capclass="token punctuation">.max_send_sge class="token operator">= attrclass="token punctuation">.capclass="token punctuation">.max_recv_sge class="token operator">= class="token number">1class="token punctuation">;
	attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">= class="token number">16class="token punctuation">;
	attrclass="token punctuation">.qp_context class="token operator">= idclass="token punctuation">;
	attrclass="token punctuation">.sq_sig_all class="token operator">= class="token number">1class="token punctuation">;
	ret class="token operator">= class="token function">rdma_create_epclass="token punctuation">(class="token operator">&idclass="token punctuation">, resclass="token punctuation">, class="token constant">NULLclass="token punctuation">, class="token operator">&attrclass="token punctuation">)class="token punctuation">;
	class="token comment">// Check to see if we got inline data allowed or not
	class="token keyword">if class="token punctuation">(attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">>= class="token number">16class="token punctuation">)
		send_flags class="token operator">= IBV_SEND_INLINEclass="token punctuation">;
	class="token keyword">else
		class="token function">printfclass="token punctuation">(class="token string">"rdma_client: device doesn't support IBV_SEND_INLINE, "
		       class="token string">"using sge sends\n"class="token punctuation">)class="token punctuation">;

	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_create_ep"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_free_addrinfoclass="token punctuation">;
	class="token punctuation">}

	mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, recv_msgclass="token punctuation">, class="token number">16class="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(class="token operator">!mrclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for recv_msg"class="token punctuation">)class="token punctuation">;
		ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
		class="token keyword">goto out_destroy_epclass="token punctuation">;
	class="token punctuation">}
	class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">) class="token punctuation">{
		send_mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, send_msgclass="token punctuation">, class="token number">16class="token punctuation">)class="token punctuation">;
		class="token keyword">if class="token punctuation">(class="token operator">!send_mrclass="token punctuation">) class="token punctuation">{
			class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for send_msg"class="token punctuation">)class="token punctuation">;
			ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
			class="token keyword">goto out_dereg_recvclass="token punctuation">;
		class="token punctuation">}
	class="token punctuation">}

	ret class="token operator">= class="token function">rdma_post_recvclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, recv_msgclass="token punctuation">, class="token number">16class="token punctuation">, mrclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_recv"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_dereg_sendclass="token punctuation">;
	class="token punctuation">}

	ret class="token operator">= class="token function">rdma_connectclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_connect"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_dereg_sendclass="token punctuation">;
	class="token punctuation">}


	class="token function">printfclass="token punctuation">(class="token string">"client send: %s\n"class="token punctuation">, class="token punctuation">(class="token keyword">char class="token operator">*class="token punctuation">)send_msgclass="token punctuation">)class="token punctuation">;
	ret class="token operator">= class="token function">rdma_post_sendclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, send_msgclass="token punctuation">, class="token number">16class="token punctuation">, send_mrclass="token punctuation">, send_flagsclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_send"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_disconnectclass="token punctuation">;
	class="token punctuation">}

	class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_send_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_send_comp"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_disconnectclass="token punctuation">;
	class="token punctuation">}

	class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_recv_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">)
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_recv_comp"class="token punctuation">)class="token punctuation">;
	class="token keyword">else
		ret class="token operator">= class="token number">0class="token punctuation">;
	
	class="token function">printfclass="token punctuation">(class="token string">"client received: %s\n"class="token punctuation">, class="token punctuation">(class="token keyword">char class="token operator">*class="token punctuation">) recv_msgclass="token punctuation">)class="token punctuation">;

out_disconnectclass="token operator">:
	class="token function">rdma_disconnectclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_dereg_sendclass="token operator">:
	class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
		class="token function">rdma_dereg_mrclass="token punctuation">(send_mrclass="token punctuation">)class="token punctuation">;
out_dereg_recvclass="token operator">:
	class="token function">rdma_dereg_mrclass="token punctuation">(mrclass="token punctuation">)class="token punctuation">;
out_destroy_epclass="token operator">:
	class="token function">rdma_destroy_epclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_free_addrinfoclass="token operator">:
	class="token function">rdma_freeaddrinfoclass="token punctuation">(resclass="token punctuation">)class="token punctuation">;
outclass="token operator">:
	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}

class="token keyword">int class="token function">mainclass="token punctuation">(class="token keyword">int argcclass="token punctuation">, class="token keyword">char class="token operator">*class="token operator">*argvclass="token punctuation">)
class="token punctuation">{
	class="token keyword">int retclass="token punctuation">;

	class="token keyword">char class="token operator">*s class="token operator">= class="token string">"hello world"class="token punctuation">;
	class="token comment">// printf("client send: %s\n", s);
	class="token function">memcpyclass="token punctuation">(send_msgclass="token punctuation">, s class="token punctuation">, class="token function">strlenclass="token punctuation">(sclass="token punctuation">)class="token punctuation">)class="token punctuation">;

	class="token function">printfclass="token punctuation">(class="token string">"rdma_client: start\n"class="token punctuation">)class="token punctuation">;
	ret class="token operator">= class="token function">runclass="token punctuation">(class="token punctuation">)class="token punctuation">;
	class="token function">printfclass="token punctuation">(class="token string">"rdma_client: end %d\n"class="token punctuation">, retclass="token punctuation">)class="token punctuation">;
	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}
code>

server端代码

<code class="prism language-c">class="token comment">// server1.c
class="token comment">/*
 * Copyright (c) 2005-2009 Intel Corporation.  All rights reserved.
 *
 * This software is available to you under the OpenIB.org BSD license
 * below:
 *
 *     Redistribution and use in source and binary forms, with or
 *     without modification, are permitted provided that the following
 *     conditions are met:
 *
 *      - Redistributions of source code must retain the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer.
 *
 *      - Redistributions in binary form must reproduce the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer in the documentation and/or other materials
 *        provided with the distribution.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
 * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
 * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
 * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */

class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdio.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdlib.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><string.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><errno.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><getopt.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><netdb.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_cma.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_verbs.h>

class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*server class="token operator">= class="token string">"0.0.0.0"class="token punctuation">;
class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*port class="token operator">= class="token string">"7471"class="token punctuation">;

class="token keyword">static class="token keyword">struct class="token class-name">rdma_cm_id class="token operator">*listen_idclass="token punctuation">, class="token operator">*idclass="token punctuation">;
class="token keyword">static class="token keyword">struct class="token class-name">ibv_mr class="token operator">*mrclass="token punctuation">, class="token operator">*send_mrclass="token punctuation">;
class="token keyword">static class="token keyword">int send_flagsclass="token punctuation">;
class="token keyword">static class="token class-name">uint8_t send_msgclass="token punctuation">[class="token number">16class="token punctuation">]class="token punctuation">;
class="token keyword">static class="token class-name">uint8_t recv_msgclass="token punctuation">[class="token number">16class="token punctuation">]class="token punctuation">;

class="token keyword">static class="token keyword">int class="token function">runclass="token punctuation">(class="token keyword">voidclass="token punctuation">)
class="token punctuation">{
	class="token keyword">struct class="token class-name">rdma_addrinfo hintsclass="token punctuation">, class="token operator">*resclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_qp_init_attr init_attrclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_qp_attr qp_attrclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_wc wcclass="token punctuation">;
	class="token keyword">int retclass="token punctuation">;

	class="token function">memsetclass="token punctuation">(class="token operator">&hintsclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof hintsclass="token punctuation">)class="token punctuation">;
	hintsclass="token punctuation">.ai_flags class="token operator">= RAI_PASSIVEclass="token punctuation">;
	hintsclass="token punctuation">.ai_port_space class="token operator">= class="tags" href="/tags/RDMA.html" title=RDMA>RDMA_PS_TCPclass="token punctuation">;
	ret class="token operator">= class="token function">rdma_getaddrinfoclass="token punctuation">(serverclass="token punctuation">, portclass="token punctuation">, class="token operator">&hintsclass="token punctuation">, class="token operator">&resclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">printfclass="token punctuation">(class="token string">"rdma_getaddrinfo: %s\n"class="token punctuation">, class="token function">gai_strerrorclass="token punctuation">(retclass="token punctuation">)class="token punctuation">)class="token punctuation">;
		class="token keyword">return retclass="token punctuation">;
	class="token punctuation">}

	class="token function">memsetclass="token punctuation">(class="token operator">&init_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof init_attrclass="token punctuation">)class="token punctuation">;
	init_attrclass="token punctuation">.capclass="token punctuation">.max_send_wr class="token operator">= init_attrclass="token punctuation">.capclass="token punctuation">.max_recv_wr class="token operator">= class="token number">1class="token punctuation">;
	init_attrclass="token punctuation">.capclass="token punctuation">.max_send_sge class="token operator">= init_attrclass="token punctuation">.capclass="token punctuation">.max_recv_sge class="token operator">= class="token number">1class="token punctuation">;
	init_attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">= class="token number">16class="token punctuation">;
	init_attrclass="token punctuation">.sq_sig_all class="token operator">= class="token number">1class="token punctuation">;
	ret class="token operator">= class="token function">rdma_create_epclass="token punctuation">(class="token operator">&listen_idclass="token punctuation">, resclass="token punctuation">, class="token constant">NULLclass="token punctuation">, class="token operator">&init_attrclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_create_ep"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_free_addrinfoclass="token punctuation">;
	class="token punctuation">}

	ret class="token operator">= class="token function">rdma_listenclass="token punctuation">(listen_idclass="token punctuation">, class="token number">0class="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_listen"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_destroy_listen_epclass="token punctuation">;
	class="token punctuation">}

	ret class="token operator">= class="token function">rdma_get_requestclass="token punctuation">(listen_idclass="token punctuation">, class="token operator">&idclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_request"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_destroy_listen_epclass="token punctuation">;
	class="token punctuation">}

	class="token function">memsetclass="token punctuation">(class="token operator">&qp_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof qp_attrclass="token punctuation">)class="token punctuation">;
	class="token function">memsetclass="token punctuation">(class="token operator">&init_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof init_attrclass="token punctuation">)class="token punctuation">;
	ret class="token operator">= class="token function">ibv_query_qpclass="token punctuation">(idclass="token operator">->qpclass="token punctuation">, class="token operator">&qp_attrclass="token punctuation">, IBV_QP_CAPclass="token punctuation">,
			   class="token operator">&init_attrclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"ibv_query_qp"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_destroy_accept_epclass="token punctuation">;
	class="token punctuation">}
	class="token keyword">if class="token punctuation">(init_attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">>= class="token number">16class="token punctuation">)
		send_flags class="token operator">= IBV_SEND_INLINEclass="token punctuation">;
	class="token keyword">else
		class="token function">printfclass="token punctuation">(class="token string">"rdma_server: device doesn't support IBV_SEND_INLINE, "
		       class="token string">"using sge sends\n"class="token punctuation">)class="token punctuation">;

	mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, recv_msgclass="token punctuation">, class="token number">16class="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(class="token operator">!mrclass="token punctuation">) class="token punctuation">{
		ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for recv_msg"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_destroy_accept_epclass="token punctuation">;
	class="token punctuation">}
	class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">) class="token punctuation">{
		send_mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, send_msgclass="token punctuation">, class="token number">16class="token punctuation">)class="token punctuation">;
		class="token keyword">if class="token punctuation">(class="token operator">!send_mrclass="token punctuation">) class="token punctuation">{
			ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
			class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for send_msg"class="token punctuation">)class="token punctuation">;
			class="token keyword">goto out_dereg_recvclass="token punctuation">;
		class="token punctuation">}
	class="token punctuation">}

	ret class="token operator">= class="token function">rdma_post_recvclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, recv_msgclass="token punctuation">, class="token number">16class="token punctuation">, mrclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_recv"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_dereg_sendclass="token punctuation">;
	class="token punctuation">}

	ret class="token operator">= class="token function">rdma_acceptclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_accept"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_dereg_sendclass="token punctuation">;
	class="token punctuation">}

	class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_recv_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_recv_comp"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_disconnectclass="token punctuation">;
	class="token punctuation">}

    class="token function">printfclass="token punctuation">(class="token string">"server received: %s\n" class="token punctuation">, class="token punctuation">(class="token keyword">char class="token operator">*class="token punctuation">)recv_msgclass="token punctuation">)class="token punctuation">;
    
    class="token keyword">char class="token operator">*s class="token operator">= class="token string">"ACK"class="token punctuation">;
    class="token function">memcpyclass="token punctuation">(send_msgclass="token punctuation">, sclass="token punctuation">, class="token function">strlenclass="token punctuation">(sclass="token punctuation">)class="token punctuation">)class="token punctuation">;
    class="token function">printfclass="token punctuation">(class="token string">"server send: %s\n"class="token punctuation">, class="token punctuation">(class="token keyword">char class="token operator">*class="token punctuation">)send_msgclass="token punctuation">)class="token punctuation">;

	ret class="token operator">= class="token function">rdma_post_sendclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, send_msgclass="token punctuation">, class="token number">16class="token punctuation">, send_mrclass="token punctuation">, send_flagsclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_send"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_disconnectclass="token punctuation">;
	class="token punctuation">}

	class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_send_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">)
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_send_comp"class="token punctuation">)class="token punctuation">;
	class="token keyword">else
		ret class="token operator">= class="token number">0class="token punctuation">;

out_disconnectclass="token operator">:
	class="token function">rdma_disconnectclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_dereg_sendclass="token operator">:
	class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
		class="token function">rdma_dereg_mrclass="token punctuation">(send_mrclass="token punctuation">)class="token punctuation">;
out_dereg_recvclass="token operator">:
	class="token function">rdma_dereg_mrclass="token punctuation">(mrclass="token punctuation">)class="token punctuation">;
out_destroy_accept_epclass="token operator">:
	class="token function">rdma_destroy_epclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_destroy_listen_epclass="token operator">:
	class="token function">rdma_destroy_epclass="token punctuation">(listen_idclass="token punctuation">)class="token punctuation">;
out_free_addrinfoclass="token operator">:
	class="token function">rdma_freeaddrinfoclass="token punctuation">(resclass="token punctuation">)class="token punctuation">;
	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}

class="token keyword">int class="token function">mainclass="token punctuation">(class="token keyword">int argcclass="token punctuation">, class="token keyword">char class="token operator">*class="token operator">*argvclass="token punctuation">)
class="token punctuation">{
	class="token keyword">int retclass="token punctuation">;

	class="token function">printfclass="token punctuation">(class="token string">"rdma_server: start\n"class="token punctuation">)class="token punctuation">;
	ret class="token operator">= class="token function">runclass="token punctuation">(class="token punctuation">)class="token punctuation">;
	class="token function">printfclass="token punctuation">(class="token string">"rdma_server: end %d\n"class="token punctuation">, retclass="token punctuation">)class="token punctuation">;
	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}
code>

首先make编译完之后࿰c;在server端执行 ./server1࿰c;然后在客户端执行./client1
运行结果：
c="https://img-blog.csdnimg.cn/direct/7962bb5b288d479fab6e4b58485d2f39.png" alt="在这里插入图片描述" />
c="https://img-blog.csdnimg.cn/direct/8204114503724d9c922516063b56e8c3.png" alt="在这里插入图片描述" />
可以看到 client向server发送了hello world࿰c;server收到之后打印出来并回复给client端ACK消息࿰c;client收到之后并打印。最后双方断开连接࿰c;完成！

version2-客户端发送一次࿰c;服务端循环等待

client2的代码跟上面一样࿰c;server2代码不一样。
server2的逻辑：在run函数进来之后记录一个connect点࿰c;当远程客户端发送完信息后࿰c;释放连接的资源࿰c;跳转到connect阶段准备让下一个client连接。

<code class="prism language-c">class="token comment">/*
 * Copyright (c) 2005-2009 Intel Corporation.  All rights reserved.
 *
 * This software is available to you under the OpenIB.org BSD license
 * below:
 *
 *     Redistribution and use in source and binary forms, with or
 *     without modification, are permitted provided that the following
 *     conditions are met:
 *
 *      - Redistributions of source code must retain the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer.
 *
 *      - Redistributions in binary form must reproduce the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer in the documentation and/or other materials
 *        provided with the distribution.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
 * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
 * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
 * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */

class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdio.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdlib.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><string.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><errno.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><getopt.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><netdb.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_cma.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_verbs.h>


class="token macro property">class="token directive-hash">#class="token directive keyword">define class="token macro-name">N class="token expression">class="token number">100
class="token macro property">class="token directive-hash">#class="token directive keyword">define class="token macro-name">MAX_CAP class="token expression">class="token number">32

class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*server class="token operator">= class="token string">"0.0.0.0"class="token punctuation">;
class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*port class="token operator">= class="token string">"7471"class="token punctuation">;

class="token keyword">static class="token keyword">struct class="token class-name">rdma_cm_id class="token operator">*listen_idclass="token punctuation">, class="token operator">*idclass="token punctuation">;
class="token keyword">static class="token keyword">struct class="token class-name">ibv_mr class="token operator">*mrclass="token punctuation">, class="token operator">*send_mrclass="token punctuation">;
class="token keyword">static class="token keyword">int send_flagsclass="token punctuation">;
class="token keyword">static class="token class-name">uint8_t send_msgclass="token punctuation">[MAX_CAPclass="token punctuation">]class="token punctuation">;
class="token keyword">static class="token class-name">uint8_t recv_msgclass="token punctuation">[MAX_CAPclass="token punctuation">]class="token punctuation">;

class="token keyword">static class="token keyword">int class="token function">runclass="token punctuation">(class="token keyword">voidclass="token punctuation">)
class="token punctuation">{
	class="token keyword">struct class="token class-name">rdma_addrinfo hintsclass="token punctuation">, class="token operator">*resclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_qp_init_attr init_attrclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_qp_attr qp_attrclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_wc wcclass="token punctuation">;
	class="token keyword">int retclass="token punctuation">;

    class="token keyword">whileclass="token punctuation">(class="token number">1class="token punctuation">)
    class="token punctuation">{

        class="token function">memsetclass="token punctuation">(class="token operator">&hintsclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof hintsclass="token punctuation">)class="token punctuation">;
        hintsclass="token punctuation">.ai_flags class="token operator">= RAI_PASSIVEclass="token punctuation">;
        hintsclass="token punctuation">.ai_port_space class="token operator">= class="tags" href="/tags/RDMA.html" title=RDMA>RDMA_PS_TCPclass="token punctuation">;
        ret class="token operator">= class="token function">rdma_getaddrinfoclass="token punctuation">(serverclass="token punctuation">, portclass="token punctuation">, class="token operator">&hintsclass="token punctuation">, class="token operator">&resclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">printfclass="token punctuation">(class="token string">"rdma_getaddrinfo: %s\n"class="token punctuation">, class="token function">gai_strerrorclass="token punctuation">(retclass="token punctuation">)class="token punctuation">)class="token punctuation">;
            class="token keyword">return retclass="token punctuation">;
        class="token punctuation">}

        class="token function">memsetclass="token punctuation">(class="token operator">&init_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof init_attrclass="token punctuation">)class="token punctuation">;
        init_attrclass="token punctuation">.capclass="token punctuation">.max_send_wr class="token operator">= init_attrclass="token punctuation">.capclass="token punctuation">.max_recv_wr class="token operator">= Nclass="token punctuation">;
        init_attrclass="token punctuation">.capclass="token punctuation">.max_send_sge class="token operator">= init_attrclass="token punctuation">.capclass="token punctuation">.max_recv_sge class="token operator">= class="token number">1class="token punctuation">;
        init_attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">= MAX_CAPclass="token punctuation">;
        init_attrclass="token punctuation">.sq_sig_all class="token operator">= class="token number">1class="token punctuation">;

        ret class="token operator">= class="token function">rdma_create_epclass="token punctuation">(class="token operator">&listen_idclass="token punctuation">, resclass="token punctuation">, class="token constant">NULLclass="token punctuation">, class="token operator">&init_attrclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_create_ep"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_free_addrinfoclass="token punctuation">;
        class="token punctuation">}

        ret class="token operator">= class="token function">rdma_listenclass="token punctuation">(listen_idclass="token punctuation">, class="token number">0class="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_listen"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_destroy_listen_epclass="token punctuation">;
        class="token punctuation">}

        ret class="token operator">= class="token function">rdma_get_requestclass="token punctuation">(listen_idclass="token punctuation">, class="token operator">&idclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_request"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_destroy_listen_epclass="token punctuation">;
        class="token punctuation">}

        class="token function">memsetclass="token punctuation">(class="token operator">&qp_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof qp_attrclass="token punctuation">)class="token punctuation">;
        class="token function">memsetclass="token punctuation">(class="token operator">&init_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof init_attrclass="token punctuation">)class="token punctuation">;
        ret class="token operator">= class="token function">ibv_query_qpclass="token punctuation">(idclass="token operator">->qpclass="token punctuation">, class="token operator">&qp_attrclass="token punctuation">, IBV_QP_CAPclass="token punctuation">,
                class="token operator">&init_attrclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"ibv_query_qp"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_destroy_accept_epclass="token punctuation">;
        class="token punctuation">}
        class="token keyword">if class="token punctuation">(init_attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">>= MAX_CAPclass="token punctuation">)
            send_flags class="token operator">= IBV_SEND_INLINEclass="token punctuation">;
        class="token keyword">else
            class="token function">printfclass="token punctuation">(class="token string">"rdma_server: device doesn't support IBV_SEND_INLINE, "
                class="token string">"using sge sends\n"class="token punctuation">)class="token punctuation">;
        mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, recv_msgclass="token punctuation">, Nclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(class="token operator">!mrclass="token punctuation">) class="token punctuation">{
            ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for recv_msg"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_destroy_accept_epclass="token punctuation">;
        class="token punctuation">}
        class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">) class="token punctuation">{
            send_mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, send_msgclass="token punctuation">, MAX_CAPclass="token punctuation">)class="token punctuation">;
            class="token keyword">if class="token punctuation">(class="token operator">!send_mrclass="token punctuation">) class="token punctuation">{
                ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
                class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for send_msg"class="token punctuation">)class="token punctuation">;
                class="token keyword">goto out_dereg_recvclass="token punctuation">;
            class="token punctuation">}
        class="token punctuation">}   
        

        ret class="token operator">= class="token function">rdma_acceptclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_accept"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_dereg_sendclass="token punctuation">;
        class="token punctuation">}

        
        class="token function">memsetclass="token punctuation">(recv_msgclass="token punctuation">, class="token number">0 class="token punctuation">, class="token keyword">sizeof recv_msgclass="token punctuation">)class="token punctuation">;
        class="token function">memsetclass="token punctuation">(send_msgclass="token punctuation">, class="token number">0 class="token punctuation">, class="token keyword">sizeof send_msgclass="token punctuation">)class="token punctuation">;
        ret class="token operator">= class="token function">rdma_post_recvclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, recv_msgclass="token punctuation">, MAX_CAPclass="token punctuation">, mrclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_recv"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_dereg_sendclass="token punctuation">;
        class="token punctuation">}
        class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_recv_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_recv_comp"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_disconnectclass="token punctuation">;
        class="token punctuation">}
        
        class="token function">printfclass="token punctuation">(class="token string">"server received: %s\n"class="token punctuation">, class="token punctuation">(class="token keyword">char class="token operator">*class="token punctuation">)recv_msgclass="token punctuation">)class="token punctuation">;
        class="token function">memcpyclass="token punctuation">(send_msgclass="token punctuation">, recv_msgclass="token punctuation">, class="token keyword">sizeofclass="token punctuation">(recv_msgclass="token punctuation">)class="token punctuation">)class="token punctuation">;
        
        ret class="token operator">= class="token function">rdma_post_sendclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, send_msgclass="token punctuation">, MAX_CAPclass="token punctuation">, send_mrclass="token punctuation">, send_flagsclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_send"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_disconnectclass="token punctuation">;
        class="token punctuation">}
        class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_send_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">; class="token comment">// 确认对方已经收到 对方会发送ack
        class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">)
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_send_comp"class="token punctuation">)class="token punctuation">;
        class="token keyword">else
            ret class="token operator">= class="token number">0class="token punctuation">;
        
        class="token function">rdma_disconnectclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
            class="token function">rdma_dereg_mrclass="token punctuation">(send_mrclass="token punctuation">)class="token punctuation">;
        class="token function">rdma_dereg_mrclass="token punctuation">(mrclass="token punctuation">)class="token punctuation">;
        class="token function">rdma_destroy_epclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
        class="token function">rdma_destroy_epclass="token punctuation">(listen_idclass="token punctuation">)class="token punctuation">;
        class="token function">rdma_freeaddrinfoclass="token punctuation">(resclass="token punctuation">)class="token punctuation">;  
    class="token punctuation">}
         
    

out_disconnectclass="token operator">:
    class="token function">rdma_disconnectclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_dereg_sendclass="token operator">:
    class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
        class="token function">rdma_dereg_mrclass="token punctuation">(send_mrclass="token punctuation">)class="token punctuation">;
out_dereg_recvclass="token operator">:
    class="token function">rdma_dereg_mrclass="token punctuation">(mrclass="token punctuation">)class="token punctuation">;
out_destroy_accept_epclass="token operator">:
    class="token function">rdma_destroy_epclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_destroy_listen_epclass="token operator">:
    class="token function">rdma_destroy_epclass="token punctuation">(listen_idclass="token punctuation">)class="token punctuation">;
out_free_addrinfoclass="token operator">:
    class="token function">rdma_freeaddrinfoclass="token punctuation">(resclass="token punctuation">)class="token punctuation">;  

	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}

class="token keyword">int class="token function">mainclass="token punctuation">(class="token keyword">int argcclass="token punctuation">, class="token keyword">char class="token operator">*class="token operator">*argvclass="token punctuation">)
class="token punctuation">{
	class="token keyword">int retclass="token punctuation">;
	class="token function">printfclass="token punctuation">(class="token string">"rdma_server: start\n"class="token punctuation">)class="token punctuation">;
	ret class="token operator">= class="token function">runclass="token punctuation">(class="token punctuation">)class="token punctuation">;
	class="token function">printfclass="token punctuation">(class="token string">"rdma_server: end %d\n"class="token punctuation">, retclass="token punctuation">)class="token punctuation">;
	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}
code>

运行结果：
c="https://img-blog.csdnimg.cn/direct/e7fe229278d043e3a82281c137b19da3.png" alt="在这里插入图片描述" />

c="https://img-blog.csdnimg.cn/direct/b5cb719b742b4fdbb08fcad65b7cb01c.png" alt="在这里插入图片描述" />

可以看到客户端发送一次消息之后便结束了࿰c;服务端却一直等待连接࿰c;直到按下ctrl+c。

version3-客户端循环发送࿰c;服务端循环等待࿰c;一次连接

和上述版本2不同的时候࿰c;这里client和server只连接一次࿰c;然后可以多次发送消息。直到client发送的消息为disconnect

<code class="prism language-c">class="token comment">// client3.c
class="token comment">/*
 * Copyright (c) 2010 Intel Corporation.  All rights reserved.
 *
 * This software is available to you under the OpenIB.org BSD license
 * below:
 *
 *     Redistribution and use in source and binary forms, with or
 *     without modification, are permitted provided that the following
 *     conditions are met:
 *
 *      - Redistributions of source code must retain the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer.
 *
 *      - Redistributions in binary form must reproduce the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer in the documentation and/or other materials
 *        provided with the distribution.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
 * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
 * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
 * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */

class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdio.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdlib.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><string.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><netdb.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><errno.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><getopt.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_cma.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_verbs.h>

class="token macro property">class="token directive-hash">#class="token directive keyword">define class="token macro-name">N class="token expression">class="token number">100
class="token macro property">class="token directive-hash">#class="token directive keyword">define class="token macro-name">MAX_CAP class="token expression">class="token number">32

class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*server class="token operator">= class="token string">"10.10.10.1"class="token punctuation">;
class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*port class="token operator">= class="token string">"7471"class="token punctuation">;

class="token keyword">static class="token keyword">struct class="token class-name">rdma_cm_id class="token operator">*idclass="token punctuation">;
class="token keyword">static class="token keyword">struct class="token class-name">ibv_mr class="token operator">*mrclass="token punctuation">, class="token operator">*send_mrclass="token punctuation">;
class="token keyword">static class="token keyword">int send_flagsclass="token punctuation">;
class="token keyword">static class="token class-name">uint8_t send_msgclass="token punctuation">[MAX_CAPclass="token punctuation">]class="token punctuation">;
class="token keyword">static class="token class-name">uint8_t recv_msgclass="token punctuation">[MAX_CAPclass="token punctuation">]class="token punctuation">;

class="token keyword">static class="token keyword">int class="token function">runclass="token punctuation">(class="token keyword">voidclass="token punctuation">)
class="token punctuation">{
	class="token keyword">struct class="token class-name">rdma_addrinfo hintsclass="token punctuation">, class="token operator">*resclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_qp_init_attr attrclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_wc wcclass="token punctuation">;
	class="token keyword">int retclass="token punctuation">;

	class="token function">memsetclass="token punctuation">(class="token operator">&hintsclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof hintsclass="token punctuation">)class="token punctuation">;
	hintsclass="token punctuation">.ai_port_space class="token operator">= class="tags" href="/tags/RDMA.html" title=RDMA>RDMA_PS_TCPclass="token punctuation">;
	ret class="token operator">= class="token function">rdma_getaddrinfoclass="token punctuation">(serverclass="token punctuation">, portclass="token punctuation">, class="token operator">&hintsclass="token punctuation">, class="token operator">&resclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">printfclass="token punctuation">(class="token string">"rdma_getaddrinfo: %s\n"class="token punctuation">, class="token function">gai_strerrorclass="token punctuation">(retclass="token punctuation">)class="token punctuation">)class="token punctuation">;
		class="token keyword">goto outclass="token punctuation">;
	class="token punctuation">}
    class="token function">memsetclass="token punctuation">(class="token operator">&attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof attrclass="token punctuation">)class="token punctuation">;
    attrclass="token punctuation">.capclass="token punctuation">.max_send_wr class="token operator">= attrclass="token punctuation">.capclass="token punctuation">.max_recv_wr class="token operator">= class="token number">5class="token punctuation">;
    attrclass="token punctuation">.capclass="token punctuation">.max_send_sge class="token operator">= attrclass="token punctuation">.capclass="token punctuation">.max_recv_sge class="token operator">= class="token number">1class="token punctuation">;
    attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">= MAX_CAPclass="token punctuation">;
    attrclass="token punctuation">.qp_context class="token operator">= idclass="token punctuation">;
    attrclass="token punctuation">.sq_sig_all class="token operator">= class="token number">1class="token punctuation">;
    ret class="token operator">= class="token function">rdma_create_epclass="token punctuation">(class="token operator">&idclass="token punctuation">, resclass="token punctuation">, class="token constant">NULLclass="token punctuation">, class="token operator">&attrclass="token punctuation">)class="token punctuation">;
    class="token comment">// Check to see if we got inline data allowed or not
    class="token keyword">if class="token punctuation">(attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">>= MAX_CAPclass="token punctuation">)
        send_flags class="token operator">= IBV_SEND_INLINEclass="token punctuation">;
    class="token keyword">else
        class="token function">printfclass="token punctuation">(class="token string">"rdma_client: device doesn't support IBV_SEND_INLINE, "
            class="token string">"using sge sends\n"class="token punctuation">)class="token punctuation">;

    class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
        class="token function">perrorclass="token punctuation">(class="token string">"rdma_create_ep"class="token punctuation">)class="token punctuation">;
        class="token keyword">goto out_free_addrinfoclass="token punctuation">;
    class="token punctuation">}

    mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, recv_msgclass="token punctuation">, MAX_CAPclass="token punctuation">)class="token punctuation">;
    class="token keyword">if class="token punctuation">(class="token operator">!mrclass="token punctuation">) class="token punctuation">{
        class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for recv_msg"class="token punctuation">)class="token punctuation">;
        ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
        class="token keyword">goto out_destroy_epclass="token punctuation">;
    class="token punctuation">}
    class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">) class="token punctuation">{
        send_mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, send_msgclass="token punctuation">, MAX_CAPclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(class="token operator">!send_mrclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for send_msg"class="token punctuation">)class="token punctuation">;
            ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
            class="token keyword">goto out_dereg_recvclass="token punctuation">;
        class="token punctuation">}
    class="token punctuation">}

    class="token comment">// ret = rdma_post_recv(id, NULL, recv_msg, 16, mr);
    class="token comment">// if (ret) {
    class="token comment">//     perror("rdma_post_recv");
    class="token comment">//     goto out_dereg_send;
    class="token comment">// }

    class="token comment">// printf("123\n");

    ret class="token operator">= class="token function">rdma_connectclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">)class="token punctuation">;
    class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
        class="token function">perrorclass="token punctuation">(class="token string">"rdma_connect"class="token punctuation">)class="token punctuation">;
        class="token keyword">goto out_dereg_sendclass="token punctuation">;
    class="token punctuation">}

    class="token keyword">whileclass="token punctuation">(class="token number">1class="token punctuation">)
    class="token punctuation">{
        class="token comment">// sleep(5);
        class="token function">memsetclass="token punctuation">(recv_msgclass="token punctuation">, class="token number">0 class="token punctuation">, class="token keyword">sizeof recv_msgclass="token punctuation">)class="token punctuation">;
        class="token function">memsetclass="token punctuation">(send_msgclass="token punctuation">, class="token number">0 class="token punctuation">, class="token keyword">sizeof send_msgclass="token punctuation">)class="token punctuation">;
        class="token function">printfclass="token punctuation">(class="token string">"input send message: "class="token punctuation">)class="token punctuation">;
        class="token function">scanfclass="token punctuation">(class="token string">"%s"class="token punctuation">, send_msgclass="token punctuation">)class="token punctuation">;
        class="token function">getcharclass="token punctuation">(class="token punctuation">)class="token punctuation">;
        
        ret class="token operator">= class="token function">rdma_post_recvclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, recv_msgclass="token punctuation">, MAX_CAPclass="token punctuation">, mrclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_recv"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_dereg_sendclass="token punctuation">;
        class="token punctuation">}

        ret class="token operator">= class="token function">rdma_post_sendclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, send_msgclass="token punctuation">, MAX_CAPclass="token punctuation">, send_mrclass="token punctuation">, send_flagsclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_send"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_disconnectclass="token punctuation">;
        class="token punctuation">}

        class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_send_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_send_comp"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_disconnectclass="token punctuation">;
        class="token punctuation">}

        class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_recv_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">)
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_recv_comp"class="token punctuation">)class="token punctuation">;
        class="token keyword">else
            ret class="token operator">= class="token number">0class="token punctuation">;
        
        class="token keyword">ifclass="token punctuation">(class="token function">strcmpclass="token punctuation">(class="token punctuation">(class="token keyword">charclass="token operator">*class="token punctuation">)send_msgclass="token punctuation">,class="token string">"disconnect"class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
        class="token punctuation">{
            class="token function">printfclass="token punctuation">(class="token string">"disconnect\n"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_disconnectclass="token punctuation">;
        class="token punctuation">}
        class="token keyword">else
        class="token punctuation">{
            class="token function">printfclass="token punctuation">(class="token string">"%s\n"class="token punctuation">, recv_msgclass="token punctuation">)class="token punctuation">;
        class="token punctuation">}
            
    class="token punctuation">}

out_disconnectclass="token operator">:
    class="token function">rdma_disconnectclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_dereg_sendclass="token operator">:
    class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
            class="token function">rdma_dereg_mrclass="token punctuation">(send_mrclass="token punctuation">)class="token punctuation">;
out_dereg_recvclass="token operator">:
    class="token function">rdma_dereg_mrclass="token punctuation">(mrclass="token punctuation">)class="token punctuation">;
out_destroy_epclass="token operator">:
    class="token function">rdma_destroy_epclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_free_addrinfoclass="token operator">:
    class="token function">rdma_freeaddrinfoclass="token punctuation">(resclass="token punctuation">)class="token punctuation">;
outclass="token operator">:
    class="token keyword">return retclass="token punctuation">;

class="token punctuation">}



class="token keyword">int class="token function">mainclass="token punctuation">(class="token keyword">int argcclass="token punctuation">, class="token keyword">char class="token operator">*class="token operator">*argvclass="token punctuation">)
class="token punctuation">{
	class="token keyword">int retclass="token punctuation">;
    class="token comment">//memcpy(send_msg, argv[1], 50);
	class="token comment">// while ((op = getopt(argc, argv, "s:p:")) != -1) {
	class="token comment">// 	switch (op) {
	class="token comment">// 	case 's':
	class="token comment">// 		server = optarg;
	class="token comment">// 		break;
	class="token comment">// 	case 'p':
	class="token comment">// 		port = optarg;
	class="token comment">// 		break;
	class="token comment">// 	default:
	class="token comment">// 		printf("usage: %s\n", argv[0]);
	class="token comment">// 		printf("\t[-s server_address]\n");
	class="token comment">// 		printf("\t[-p port_number]\n");
	class="token comment">// 		exit(1);
	class="token comment">// 	}
	class="token comment">// }

	class="token function">printfclass="token punctuation">(class="token string">"rdma_client: start\n"class="token punctuation">)class="token punctuation">;
	ret class="token operator">= class="token function">runclass="token punctuation">(class="token punctuation">)class="token punctuation">;
	class="token function">printfclass="token punctuation">(class="token string">"rdma_client: end %d\n"class="token punctuation">, retclass="token punctuation">)class="token punctuation">;
    
	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}
code>

<code class="prism language-c">class="token comment">// server3.c
class="token comment">/*
 * Copyright (c) 2005-2009 Intel Corporation.  All rights reserved.
 *
 * This software is available to you under the OpenIB.org BSD license
 * below:
 *
 *     Redistribution and use in source and binary forms, with or
 *     without modification, are permitted provided that the following
 *     conditions are met:
 *
 *      - Redistributions of source code must retain the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer.
 *
 *      - Redistributions in binary form must reproduce the above
 *        copyright notice, this list of conditions and the following
 *        disclaimer in the documentation and/or other materials
 *        provided with the distribution.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AWV
 * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
 * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
 * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 */

class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdio.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><stdlib.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><string.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><errno.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><getopt.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><netdb.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_cma.h>
class="token macro property">class="token directive-hash">#class="token directive keyword">include class="token string"><rdma/rdma_verbs.h>


class="token macro property">class="token directive-hash">#class="token directive keyword">define class="token macro-name">N class="token expression">class="token number">100
class="token macro property">class="token directive-hash">#class="token directive keyword">define class="token macro-name">MAX_CAP class="token expression">class="token number">32

class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*server class="token operator">= class="token string">"0.0.0.0"class="token punctuation">;
class="token keyword">static class="token keyword">const class="token keyword">char class="token operator">*port class="token operator">= class="token string">"7471"class="token punctuation">;

class="token keyword">static class="token keyword">struct class="token class-name">rdma_cm_id class="token operator">*listen_idclass="token punctuation">, class="token operator">*idclass="token punctuation">;
class="token keyword">static class="token keyword">struct class="token class-name">ibv_mr class="token operator">*mrclass="token punctuation">, class="token operator">*send_mrclass="token punctuation">;
class="token keyword">static class="token keyword">int send_flagsclass="token punctuation">;
class="token keyword">static class="token class-name">uint8_t send_msgclass="token punctuation">[MAX_CAPclass="token punctuation">]class="token punctuation">;
class="token keyword">static class="token class-name">uint8_t recv_msgclass="token punctuation">[MAX_CAPclass="token punctuation">]class="token punctuation">;

class="token keyword">static class="token keyword">int class="token function">runclass="token punctuation">(class="token keyword">voidclass="token punctuation">)
class="token punctuation">{
	class="token keyword">struct class="token class-name">rdma_addrinfo hintsclass="token punctuation">, class="token operator">*resclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_qp_init_attr init_attrclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_qp_attr qp_attrclass="token punctuation">;
	class="token keyword">struct class="token class-name">ibv_wc wcclass="token punctuation">;
	class="token keyword">int retclass="token punctuation">;

connectclass="token operator">:
	class="token function">memsetclass="token punctuation">(class="token operator">&hintsclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof hintsclass="token punctuation">)class="token punctuation">;
	hintsclass="token punctuation">.ai_flags class="token operator">= RAI_PASSIVEclass="token punctuation">;
	hintsclass="token punctuation">.ai_port_space class="token operator">= class="tags" href="/tags/RDMA.html" title=RDMA>RDMA_PS_TCPclass="token punctuation">;
	ret class="token operator">= class="token function">rdma_getaddrinfoclass="token punctuation">(serverclass="token punctuation">, portclass="token punctuation">, class="token operator">&hintsclass="token punctuation">, class="token operator">&resclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">printfclass="token punctuation">(class="token string">"rdma_getaddrinfo: %s\n"class="token punctuation">, class="token function">gai_strerrorclass="token punctuation">(retclass="token punctuation">)class="token punctuation">)class="token punctuation">;
		class="token keyword">return retclass="token punctuation">;
	class="token punctuation">}

	class="token function">memsetclass="token punctuation">(class="token operator">&init_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof init_attrclass="token punctuation">)class="token punctuation">;
	init_attrclass="token punctuation">.capclass="token punctuation">.max_send_wr class="token operator">= init_attrclass="token punctuation">.capclass="token punctuation">.max_recv_wr class="token operator">= Nclass="token punctuation">;
	init_attrclass="token punctuation">.capclass="token punctuation">.max_send_sge class="token operator">= init_attrclass="token punctuation">.capclass="token punctuation">.max_recv_sge class="token operator">= class="token number">1class="token punctuation">;
	init_attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">= MAX_CAPclass="token punctuation">;
	init_attrclass="token punctuation">.sq_sig_all class="token operator">= class="token number">1class="token punctuation">;

	ret class="token operator">= class="token function">rdma_create_epclass="token punctuation">(class="token operator">&listen_idclass="token punctuation">, resclass="token punctuation">, class="token constant">NULLclass="token punctuation">, class="token operator">&init_attrclass="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_create_ep"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_free_addrinfoclass="token punctuation">;
	class="token punctuation">}

	ret class="token operator">= class="token function">rdma_listenclass="token punctuation">(listen_idclass="token punctuation">, class="token number">0class="token punctuation">)class="token punctuation">;
	class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
		class="token function">perrorclass="token punctuation">(class="token string">"rdma_listen"class="token punctuation">)class="token punctuation">;
		class="token keyword">goto out_destroy_listen_epclass="token punctuation">;
	class="token punctuation">}

    ret class="token operator">= class="token function">rdma_get_requestclass="token punctuation">(listen_idclass="token punctuation">, class="token operator">&idclass="token punctuation">)class="token punctuation">;
    class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
        class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_request"class="token punctuation">)class="token punctuation">;
        class="token keyword">goto out_destroy_listen_epclass="token punctuation">;
    class="token punctuation">}

    class="token function">memsetclass="token punctuation">(class="token operator">&qp_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof qp_attrclass="token punctuation">)class="token punctuation">;
    class="token function">memsetclass="token punctuation">(class="token operator">&init_attrclass="token punctuation">, class="token number">0class="token punctuation">, class="token keyword">sizeof init_attrclass="token punctuation">)class="token punctuation">;
    ret class="token operator">= class="token function">ibv_query_qpclass="token punctuation">(idclass="token operator">->qpclass="token punctuation">, class="token operator">&qp_attrclass="token punctuation">, IBV_QP_CAPclass="token punctuation">,
            class="token operator">&init_attrclass="token punctuation">)class="token punctuation">;
    class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
        class="token function">perrorclass="token punctuation">(class="token string">"ibv_query_qp"class="token punctuation">)class="token punctuation">;
        class="token keyword">goto out_destroy_accept_epclass="token punctuation">;
    class="token punctuation">}
    class="token keyword">if class="token punctuation">(init_attrclass="token punctuation">.capclass="token punctuation">.max_inline_data class="token operator">>= MAX_CAPclass="token punctuation">)
        send_flags class="token operator">= IBV_SEND_INLINEclass="token punctuation">;
    class="token keyword">else
        class="token function">printfclass="token punctuation">(class="token string">"rdma_server: device doesn't support IBV_SEND_INLINE, "
            class="token string">"using sge sends\n"class="token punctuation">)class="token punctuation">;
    mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, recv_msgclass="token punctuation">, Nclass="token punctuation">)class="token punctuation">;
    class="token keyword">if class="token punctuation">(class="token operator">!mrclass="token punctuation">) class="token punctuation">{
        ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
        class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for recv_msg"class="token punctuation">)class="token punctuation">;
        class="token keyword">goto out_destroy_accept_epclass="token punctuation">;
    class="token punctuation">}
    class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">) class="token punctuation">{
        send_mr class="token operator">= class="token function">rdma_reg_msgsclass="token punctuation">(idclass="token punctuation">, send_msgclass="token punctuation">, MAX_CAPclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(class="token operator">!send_mrclass="token punctuation">) class="token punctuation">{
            ret class="token operator">= class="token operator">-class="token number">1class="token punctuation">;
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_reg_msgs for send_msg"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_dereg_recvclass="token punctuation">;
        class="token punctuation">}
    class="token punctuation">}   
    

    ret class="token operator">= class="token function">rdma_acceptclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">)class="token punctuation">;
    class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
        class="token function">perrorclass="token punctuation">(class="token string">"rdma_accept"class="token punctuation">)class="token punctuation">;
        class="token keyword">goto out_dereg_sendclass="token punctuation">;
    class="token punctuation">}

    class="token keyword">while class="token punctuation">(class="token number">1class="token punctuation">) class="token punctuation">{
        class="token function">memsetclass="token punctuation">(recv_msgclass="token punctuation">, class="token number">0 class="token punctuation">, class="token keyword">sizeof recv_msgclass="token punctuation">)class="token punctuation">;
        class="token function">memsetclass="token punctuation">(send_msgclass="token punctuation">, class="token number">0 class="token punctuation">, class="token keyword">sizeof send_msgclass="token punctuation">)class="token punctuation">;
        ret class="token operator">= class="token function">rdma_post_recvclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, recv_msgclass="token punctuation">, MAX_CAPclass="token punctuation">, mrclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_recv"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_dereg_sendclass="token punctuation">;
        class="token punctuation">}
        class="token keyword">while class="token punctuation">(class="token punctuation">(ret class="token operator">= class="token function">rdma_get_recv_compclass="token punctuation">(idclass="token punctuation">, class="token operator">&wcclass="token punctuation">)class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(ret class="token operator">< class="token number">0class="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_get_recv_comp"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_disconnectclass="token punctuation">;
        class="token punctuation">}
        
        class="token keyword">char class="token operator">*s class="token operator">= class="token punctuation">(class="token keyword">char class="token operator">*class="token punctuation">)recv_msgclass="token punctuation">;
        class="token keyword">int total_length class="token operator">= class="token function">strlenclass="token punctuation">(class="token string">"server get "class="token punctuation">) class="token operator">+ class="token function">strlenclass="token punctuation">(sclass="token punctuation">)class="token punctuation">; class="token comment">// 加1是为了存储字符串结束符'\0'
        class="token keyword">char class="token operator">*recv_str class="token operator">= class="token punctuation">(class="token keyword">char class="token operator">*class="token punctuation">)class="token function">mallocclass="token punctuation">(total_lengthclass="token punctuation">)class="token punctuation">;  class="token comment">// 分配足够的空间
        class="token function">strcpyclass="token punctuation">(recv_strclass="token punctuation">, class="token string">"server get "class="token punctuation">)class="token punctuation">;
        class="token function">strcatclass="token punctuation">(recv_strclass="token punctuation">, sclass="token punctuation">)class="token punctuation">;
        class="token comment">//printf("%s\n", recv_str);
        class="token function">memcpyclass="token punctuation">(send_msgclass="token punctuation">, recv_strclass="token punctuation">, class="token function">strlenclass="token punctuation">(recv_strclass="token punctuation">)class="token punctuation">)class="token punctuation">;
        
        ret class="token operator">= class="token function">rdma_post_sendclass="token punctuation">(idclass="token punctuation">, class="token constant">NULLclass="token punctuation">, send_msgclass="token punctuation">, MAX_CAPclass="token punctuation">, send_mrclass="token punctuation">, send_flagsclass="token punctuation">)class="token punctuation">;
        class="token keyword">if class="token punctuation">(retclass="token punctuation">) class="token punctuation">{
            class="token function">perrorclass="token punctuation">(class="token string">"rdma_post_send"class="token punctuation">)class="token punctuation">;
            class="token keyword">goto out_disconnectclass="token punctuation">;
        class="token punctuation">}

        class="token keyword">ifclass="token punctuation">(class="token function">strcmpclass="token punctuation">(class="token punctuation">(class="token keyword">charclass="token operator">*class="token punctuation">)recv_msgclass="token punctuation">,class="token string">"disconnect"class="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
        class="token punctuation">{
            class="token comment">//printf("%s\n",recv_msg);
            class="token function">printfclass="token punctuation">(class="token string">"client disconnect\n"class="token punctuation">)class="token punctuation">;
            class="token function">rdma_disconnectclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
            class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
                class="token function">rdma_dereg_mrclass="token punctuation">(send_mrclass="token punctuation">)class="token punctuation">;
            class="token function">rdma_dereg_mrclass="token punctuation">(mrclass="token punctuation">)class="token punctuation">;
            class="token function">rdma_destroy_epclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
            class="token function">rdma_destroy_epclass="token punctuation">(listen_idclass="token punctuation">)class="token punctuation">;
            class="token function">rdma_freeaddrinfoclass="token punctuation">(resclass="token punctuation">)class="token punctuation">;  
            class="token comment">//goto out_disconnect;
            class="token keyword">goto connectclass="token punctuation">;
            
        class="token punctuation">}
        class="token keyword">else
        class="token punctuation">{
            class="token function">printfclass="token punctuation">(class="token string">"%s\n"class="token punctuation">, recv_msgclass="token punctuation">)class="token punctuation">;
        class="token punctuation">}
        
        class="token comment">// while ((ret = rdma_get_send_comp(id, &wc)) == 0); // 确认对方已经收到 对方发送ack
        class="token comment">// printf("after send\n");
        class="token comment">// if (ret < 0)
        class="token comment">//     perror("rdma_get_send_comp");
        class="token comment">// else
		class="token comment">//     ret = 0;
         
    class="token punctuation">}

out_disconnectclass="token operator">:
    class="token function">rdma_disconnectclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_dereg_sendclass="token operator">:
    class="token keyword">if class="token punctuation">(class="token punctuation">(send_flags class="token operator">& IBV_SEND_INLINEclass="token punctuation">) class="token operator">== class="token number">0class="token punctuation">)
        class="token function">rdma_dereg_mrclass="token punctuation">(send_mrclass="token punctuation">)class="token punctuation">;
out_dereg_recvclass="token operator">:
    class="token function">rdma_dereg_mrclass="token punctuation">(mrclass="token punctuation">)class="token punctuation">;
out_destroy_accept_epclass="token operator">:
    class="token function">rdma_destroy_epclass="token punctuation">(idclass="token punctuation">)class="token punctuation">;
out_destroy_listen_epclass="token operator">:
    class="token function">rdma_destroy_epclass="token punctuation">(listen_idclass="token punctuation">)class="token punctuation">;
out_free_addrinfoclass="token operator">:
    class="token function">rdma_freeaddrinfoclass="token punctuation">(resclass="token punctuation">)class="token punctuation">;  

	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}

class="token keyword">int class="token function">mainclass="token punctuation">(class="token keyword">int argcclass="token punctuation">, class="token keyword">char class="token operator">*class="token operator">*argvclass="token punctuation">)
class="token punctuation">{
	class="token keyword">int opclass="token punctuation">, retclass="token punctuation">;

	class="token keyword">while class="token punctuation">(class="token punctuation">(op class="token operator">= class="token function">getoptclass="token punctuation">(argcclass="token punctuation">, argvclass="token punctuation">, class="token string">"s:p:"class="token punctuation">)class="token punctuation">) class="token operator">!= class="token operator">-class="token number">1class="token punctuation">) class="token punctuation">{
		class="token keyword">switch class="token punctuation">(opclass="token punctuation">) class="token punctuation">{
		class="token keyword">case class="token char">'s'class="token operator">:
			server class="token operator">= optargclass="token punctuation">;
			class="token keyword">breakclass="token punctuation">;
		class="token keyword">case class="token char">'p'class="token operator">:
			port class="token operator">= optargclass="token punctuation">;
			class="token keyword">breakclass="token punctuation">;
		class="token keyword">defaultclass="token operator">:
			class="token function">printfclass="token punctuation">(class="token string">"usage: %s\n"class="token punctuation">, argvclass="token punctuation">[class="token number">0class="token punctuation">]class="token punctuation">)class="token punctuation">;
			class="token function">printfclass="token punctuation">(class="token string">"\t[-s server_address]\n"class="token punctuation">)class="token punctuation">;
			class="token function">printfclass="token punctuation">(class="token string">"\t[-p port_number]\n"class="token punctuation">)class="token punctuation">;
			class="token function">exitclass="token punctuation">(class="token number">1class="token punctuation">)class="token punctuation">;
		class="token punctuation">}
	class="token punctuation">}

	class="token function">printfclass="token punctuation">(class="token string">"rdma_server: start\n"class="token punctuation">)class="token punctuation">;
	ret class="token operator">= class="token function">runclass="token punctuation">(class="token punctuation">)class="token punctuation">;
	class="token function">printfclass="token punctuation">(class="token string">"rdma_server: end %d\n"class="token punctuation">, retclass="token punctuation">)class="token punctuation">;
	class="token keyword">return retclass="token punctuation">;
class="token punctuation">}
code>

运行结果：

c="https://img-blog.csdnimg.cn/direct/76802b77390d49408f8ad6c7f36f231b.png" alt="在这里插入图片描述" />
c="https://img-blog.csdnimg.cn/direct/f25ab9d197df4d3aa1e2d2220ce4587b.png" alt="在这里插入图片描述" />

总结：

本文实现了rdma中class="tags" href="/tags/SEND-RECEIVE.html" title=send-receive>send-receive双边原语的三种需求版本࿰c;从单次发送到两者都能多次发送。理解其中的代码逻辑࿰c;想要发送消息之前对端得创建一个recv队列用来接收消息。发送完了有一个发送完成队列࿰c;接收完了也有一个接收完成队列。最后双方断开连接需要一起断开࿰c;不能某一方执行disconnect另一方不执行。本次实验有一个关键点：

<code>while ((ret = rdma_get_send_comp(id, &wc)) == 0)
code>

这一行代码是等待发送成功࿰c;发送成功之后࿰c;对方会给一个隐式信息表示我已经收到。这里耗费得时间比较长一点࿰c;在版本3中࿰c;如果不注释掉࿰c;在server端的receive队列还没有建立好࿰c;这就导致client发送了消息࿰c;server还没有收到࿰c;双方就陷入了死循环中。