Darknet modifications
The following are the changes I made to the darknet neural network framework for use with my Izzy, artificial intelligence robot. The changes were so that:
- I could communicate with darknet using Unix Domain Sockets (UDS) to send it commands via write() and get results back via read(), and
- I could give darknet the image in shared memory and get a modified version of the image back in the same shared memory. The modified version has any recognized objects labeled and enclosed in a bounding box.
How I run darknet
The commands I give on the command line to run darknet are:
sudo rm /dnserver sudo ./darknet detect cfg/yolov3-tiny.cfg yolov3-tiny.weights sock=/dnserver
The rm /dnserver is because that's the name I use in the sample command line above with the sock option. It's the name for the socket that UDS creates in the pathname space. It needs to be deleted before running darknet again.
How to communicate with the changed darknet
In the darknet command line above you can see that I'm giving it the detect option followed by a configuration file, a weights file and sock={socket_name}. The sock={socket_name} is my first change. Normally you give it the name of a file containing an image to do object recognition with. Giving it sock={socket_name} tells darknet to instead create an endpoint for the communication (socket()), bind the sock_name to it, then listen for it and finally accept a connection. It then does a read() to wait for your commands from over that socket.
The following are the commands that darknet expects over that socket. You send it those commands using write().
Command - Tell darknet about your shared memory
The first thing you would do is to create a POSIX named shared memory object and send darknet the following shm= command to tell darknet about your shared memory.
shm={shared_memory_object_name}:w={width}:h={height}
darknet then opens the shared memory object of the size width*height*3, the 3 referring to 3 bytes, one for red, one for green and one for blue. You'd normally send the shm= command with w= and h= only once. If you give it again with or without the same name then darknet treats it as a new shared memory object.
Command - Give updated images for darknet to process
From then on you'd put a new image in the shared memory and send darknet the following shm= command to tell it that the image has been updated and that darknet should process it.
shm={shared_memory_object_name}:updated
darknet then runs the image through the neural network. To get the results you follow writing of the above update command by a read(). When the read() returns, darknet will have modified the image in the shared memory such that any objects it recognized are now labeled and have a rectangle around them. The results from the read() are a string of zero or more '\n' terminated lines of the format below. After the last line is a byte containing '\0'. If no objects were recognized then the first byte of the string is '\0'.
{object_label}:{probability}:{left_x}:{right_x}:{top_y}:{bottom_y}\n
Download darknet
To download darknet, go here. I found that page by starting on the Yolo 3 webpage so if you're interested in using the Yolo neural network with darknet then start there. There doesn't seem to be any version for darknet so the best I can say is that I downloaded it in February of 2022.
The changes
The files I changed are as follows. The details for each file are below.
- examples/detector.c
- src/image.c
- include/darknet.h
- Makefile
The files are available by downloading the following:
darknet_changed_files_20221119.zip
In the code below, the code in red was what was added. Any code in black was already there and is shown for reference to help you find where the code was added.
examples/detector.c
The following was added near the top of the file.
#include "darknet.h" // things needed for the communication code #include <errno.h> #include <unistd.h> // for the UNIX domain socket communication #include <sys/socket.h> #include <sys/un.h> // for the shared memory #include <sys/mman.h> #include <fcntl.h>
The following functions all beginning with the text, comm_, were added to support the communication over sockets. It was all added immediately before the test_detector() function.
struct shm_obj;
typedef struct shm_obj {
char *name;
int w;
int h;
unsigned char *data;
struct shm_obj *next;
} shm_obj_t;
shm_obj_t *shmlist = NULL;
static shm_obj_t *comm_add_shm_obj(char *name, int w, int h)
{
shm_obj_t *tshm;
if((tshm = malloc(sizeof(shm_obj_t))) == NULL){
fprintf(stderr, "No memory\n");
return NULL;
}
if((tshm->name = malloc(strlen(name)+1)) == NULL){
free(tshm);
fprintf(stderr, "No memory\n");
return NULL;
}
strcpy(tshm->name, name);
tshm->w = w;
tshm->h = h;
tshm->data = NULL;
tshm->next = shmlist;
shmlist = tshm;
return tshm;
}
static shm_obj_t *comm_lookup_shm_obj(char *name)
{
shm_obj_t *tshm;
tshm = shmlist;
while (tshm != NULL){
if(strcmp(tshm->name, name) == 0){
return tshm;
}
tshm = tshm->next;
}
return NULL;
}
static void comm_delete_shm_obj(shm_obj_t *shm)
{
shm_obj_t *tshm, *pshm = NULL;
tshm = shmlist;
while (tshm != NULL){
if(strcmp(tshm->name, shm->name) == 0){
if (pshm){ // if not the first one
pshm->next = tshm->next;
} else { // it's the first one
shmlist = tshm->next;
}
free(tshm->name);
free(tshm);
return;
}
pshm = tshm;
tshm = tshm->next;
}
}
#define COMM_RET_NEW_SHM 0
#define COMM_RET_UPDATED_IMG 1
// TODO: The following should be in a common communication header used
// here and by the client
#define COMM_BUFMAX_NEW_SHM (NAME_MAX+10+5+5+1)
#define COMM_BUFMAX_UPDATED_IMG (NAME_MAX+12+1)
#define COMM_BUFMAX_MAX COMM_BUFMAX_NEW_SHM // the lagest one
#define COMM_BUFMAX_DETECTION 512
/*
* comm_parse_and_process_shm - We got a string beginning with "shm=",
* now parse and process the text that comes after it.
*
* COMM_RET_NEW_SHM "shm=<shared_memory_object_name>:w=<width>:h=<height>"
* This means a client has connected and you should open the given
* shared memory object and map it in with a size of w*h*3.
* Continue back to the top of the loop (which gets back to the
* above waiting on a read).
* Note that giving the same name for one that has previously been given
* replaces the previous one.
* COMM_RET_UPDATED_IMG "shm=<shared_memory_object_name>:updated"
* This is notification that there's new data in the specified shared
* memory object.
*
* The string given in c will be modified by this function.
*
* Returns:
* COMM_* - see the above description, shm points to the object
* -1 - error
*/
static int comm_parse_and_process_shm(char *c, shm_obj_t **shm)
{
int shmfd;
char *p, *shmname;
shm_obj_t *tshm;
unsigned char *ptr;
// get the shared memory object name
if((p = strchr(c, ':')) == NULL || *p == '\0')
goto invalid_cmd;
*p = '\0';
if (strlen(c) > NAME_MAX) {
fprintf(stderr, "Shared memory object name too long (max %d)\n",
NAME_MAX);
return -1;
}
shmname = c;
p++;
if (strncmp("w=", p, 2) == 0){
int w, h;
c = (p + 2);
if(*c == ':' || (p = strchr(c, ':')) == NULL || *p == '\0')
goto invalid_cmd;
*p = '\0';
w = atoi(c);
c = (p+1);
if(strncmp("h=", c, 2) != 0)
goto invalid_cmd;
c += 2;
if (*c == '\0')
goto invalid_cmd;
h = atoi(c);
tshm = comm_lookup_shm_obj(shmname);
if (tshm) // if an existing one has this name the replace it
comm_delete_shm_obj(tshm);
if ((tshm = comm_add_shm_obj(shmname, w, h)) == NULL)
return -1;
if((shmfd = shm_open(shmname, O_RDWR, 0)) == -1){
fprintf(stderr, "Opening shared memory %s failed: %s\n",
shmname, strerror(errno));
comm_delete_shm_obj(tshm);
return -1;
}
// length calculation assumes 3 bytes per pixel and 0 stride
ptr = mmap(0, w*h*3, PROT_READ|PROT_WRITE, MAP_SHARED, shmfd, 0);
close(shmfd);
if(ptr == MAP_FAILED){
fprintf(stderr, "Mapping shared memory %s failed: %s\n",
shmname, strerror(errno));
comm_delete_shm_obj(tshm);
return -1;
}
tshm->data = ptr;
tshm->w = w;
tshm->h = h;
*shm = tshm;
return COMM_RET_NEW_SHM;
} else if (strncmp("updated", p, 7) == 0){
if ((tshm = comm_lookup_shm_obj(shmname)) == NULL) {
fprintf(stderr, "Error: asked to update non-existent shared memory object name: %s\n", shmname);
return -1;
}
*shm = tshm;
return COMM_RET_UPDATED_IMG;
}
invalid_cmd:
fprintf(stderr, "Error in format or content of communication command "
"containing at least %s\n", c);
return -1;
}
static int comm_create_and_wait_for_socket(char *sockname)
{
int sockfd, clientfd;
struct sockaddr_un saddr, peer_addr;
socklen_t peer_addr_len;
// create an endpoint for the communication
// domain=AF_LOCAL (synonym for AF_UNIX)
// type=SOCK_STREAM for TCP
// procotol=0 since only one protocol exists
if((sockfd = socket(AF_LOCAL, SOCK_STREAM, 0)) == -1){
perror("socket()");
return -1;
}
// bind a name to the socket
memset(&saddr, 0, sizeof(saddr));
saddr.sun_family = AF_LOCAL;
strncpy(saddr.sun_path, sockname, sizeof(saddr.sun_path)-1);
if(bind(sockfd, (struct sockaddr *) &saddr, sizeof(saddr)) == -1){
perror("bind()");
return -1;
}
// mark the socket as a passive socket, one that can wait for
// a client to connect()
if(listen(sockfd, 50) == -1){
perror("listen()");
return -1;
}
// wait for a connection and extract the first connection
// request on the queue. return a new file descriptor
// referring to the socket
peer_addr_len = sizeof(peer_addr);
if((clientfd = accept(sockfd, (struct sockaddr *) &peer_addr,
&peer_addr_len)) == -1){
perror("accept()");
return -1;
}
return clientfd;
}
/*
* comm_make_detections_string
*
* For each detection, dets[i], there may be multiple candidates for
* what it is. dets[i].prob is an array of size "classes". Each
* dets[i].prob[j] is the probability for that class, j. For each
* detection, we care about only the class that has the highest probability
* of being right, provided that it meets our threshold.
*
*/
void comm_make_detections_string(char *buf, int buflen, image im, detection *dets, int num, float thresh, char **names, int classes)
{
int i,j,maxj=0,len;
float maxprob;
buf[0] = '\0';
for(i = 0; i < num; ++i){
maxprob = 0.0;
for(j = 0; j < classes; ++j){
if (dets[i].prob[j] > thresh && dets[i].prob[j] > maxprob){
maxprob = dets[i].prob[j];
maxj = j;
}
}
if(maxprob > 0.0){
box b = dets[i].bbox;
//printf("%f %f %f %f\n", b.x, b.y, b.w, b.h);
int left = (b.x-b.w/2.)*im.w;
int right = (b.x+b.w/2.)*im.w;
int top = (b.y-b.h/2.)*im.h;
int bot = (b.y+b.h/2.)*im.h;
if(left < 0) left = 0;
if(right > im.w-1) right = im.w-1;
if(top < 0) top = 0;
if(bot > im.h-1) bot = im.h-1;
len = strlen(buf);
snprintf(&buf[len], buflen-len,
"%s:%.2f:%d:%d:%d:%d\n",
names[maxj], dets[i].prob[maxj], left, right, top, bot);
}
}
}
void test_detector(char *datacfg, char *cfgfile, char *weightfile, char *filename, float thresh, float hier_thresh, char *outfile, int fullscreen)
{
The following is test_detector() where the comm_ functions are called for doing the UDS communication and where the object recognition is done along with drawing any labels and rectangles on the image.
Note that due to the adding of the if(strncmp("sock=", filename, 5) == 0){, I ended up restructuring the if/else statements and so had to call load_image_color() twice instead of once.
void test_detector(char *datacfg, char *cfgfile, char *weightfile, char *filename, float thresh, float hier_thresh, char *outfile, int fullscreen)
{
list *options = read_data_cfg(datacfg);
char *name_list = option_find_str(options, "names", "data/names.list");
char **names = get_labels(name_list);
char *sockname = NULL;
int clientfd;
shm_obj_t *shm = NULL; // to elminate compiler warning about non-use
char buf[COMM_BUFMAX_MAX], dbuf[COMM_BUFMAX_DETECTION];
image **alphabet = load_alphabet();
network *net = load_network(cfgfile, weightfile, 0);
set_batch_network(net, 1);
srand(2222222);
double time;
char buff[256];
char *input = buff;
float nms=.45;
while(1){
image im;
if(filename){
if(strncmp("sock=", filename, 5) == 0){
// TODO: delete socket from Linux filesystem on exit
if(sockname == NULL){ // first time
if (strlen(&filename[5]) > 107) {
fprintf(stderr, "Error: sock= command line argument path too long (max 107).\n");
return;
}
sockname = &filename[5];
//printf("sockname=%s\n", sockname);
printf("Waiting for socket connection from socket %s...\n", sockname);
if((clientfd = comm_create_and_wait_for_socket(sockname)) == -1)
return;
}
// wait for commands
if (read(clientfd, buf, sizeof(buf)) == -1) {
perror("Socket read for image notification failed\n");
return;
}
if(strncmp("shm=", buf, 3) == 0){
switch (comm_parse_and_process_shm(&buf[4], &shm)){
case COMM_RET_NEW_SHM:
// opened a new shared memory object
// shm=<shared_memory_object_name>:w=<width>:h=<height>
// go back and wait for (read()) another command
continue;
case COMM_RET_UPDATED_IMG:
// the image in the memory has been updated
// shm=<shared_memory_object_name>:updated
strncpy(input, sockname, 256);
im = load_image_from_data(shm->data, shm->w, shm->h);
if (im.data == NULL) {
fprintf(stderr, "No memory\n");
return;
}
break;
case -1:
// fall through
default:
fprintf(stderr, "Error parsing or setting up shm= command.\n");
return;
}
}
} else {
strncpy(input, filename, 256);
im = load_image_color(input,0,0);
}
} else {
printf("Enter Image Path: ");
fflush(stdout);
input = fgets(input, 256, stdin);
if(!input) return;
strtok(input, "\n");
im = load_image_color(input,0,0);
}
image sized = letterbox_image(im, net->w, net->h);
//image sized = resize_image(im, net->w, net->h);
//image sized2 = resize_max(im, net->w);
//image sized = crop_image(sized2, -((net->w - sized2.w)/2), -((net->h - sized2.h)/2), net->w, net->h);
//resize_network(net, sized.w, sized.h);
layer l = net->layers[net->n-1];
float *X = sized.data;
time=what_time_is_it_now();
network_predict(net, X);
printf("%s: Predicted in %f seconds.\n", input, what_time_is_it_now()-time);
int nboxes = 0;
detection *dets = get_network_boxes(net, im.w, im.h, thresh, hier_thresh, 0, 1, &nboxes);
printf("%d\n", nboxes);
//if (nms) do_nms_obj(boxes, probs, l.w*l.h*l.n, l.classes, nms);
if (nms) do_nms_sort(dets, nboxes, l.classes, nms);
draw_detections(im, dets, nboxes, thresh, names, alphabet, l.classes);
if (sockname) {
if (nboxes > 0) {
comm_make_detections_string(dbuf, sizeof(dbuf)-1, im, dets, nboxes, thresh, names, l.classes);
if (buf[0] != '\0') {
printf("%s\n", dbuf);
// comm_make_detections_string() drew rectangle(s) on the image,
// put it in the shared memory
copy_image_to_data(shm->data, &im, im.w, im.h);
if (write(clientfd, dbuf, strlen(dbuf)+1) == -1)
perror("Writing detection to socket");
}
} else {
printf("Nothing detected\n");
if (write(clientfd, "", 1) == -1)
perror("Writing NULL detection to socket");
}
}
free_detections(dets, nboxes);
if(outfile){
save_image(im, outfile);
}
else{
save_image(im, "predictions");
#ifdef OPENCV
make_window("predictions", 512, 512, 0);
show_image(im, "predictions", 0);
#endif
}
free_image(im);
free_image(sized);
if (filename && sockname == NULL) break;
}
}
src/image.c
The following two functions were added between the load_image_color() and the get_image_layer() functions. They are called from the code that was added to the above test_detector() function.
image load_image_from_data(unsigned char *data, int w, int h)
{
int i,j,k;
int c = 3;
image im = make_image(w, h, c);
for(k = 0; k < c; ++k){
for(j = 0; j < h; ++j){
for(i = 0; i < w; ++i){
int dst_index = i + w*j + w*h*k;
int src_index = k + c*i + c*w*j;
im.data[dst_index] = (float)data[src_index]/255.;
}
}
}
return im;
}
void copy_image_to_data(unsigned char *data, image *im, int w, int h)
{
int i,j,k;
int c = 3;
for(k = 0; k < c; ++k){
for(j = 0; j < h; ++j){
for(i = 0; i < w; ++i){
int src_index = i + w*j + w*h*k;
int dst_index = k + c*i + c*w*j;
data[dst_index] = im->data[src_index]*255;
}
}
}
}
Originally, darknet would draw just the label of the recognized object in the image. I modified it to also draw the probability. That was done by some simple modifications to the draw_detections() function. The following shows which lines were deleted by commenting them out (shown in blue) and which lines were added (shown in red).
void draw_detections(image im, detection *dets, int num, float thresh, char **names, image **alphabet, int classes)
{
int i,j;
for(i = 0; i < num; ++i){
char labelstr[4096] = {0};
int class = -1;
int p = 0; // added
for(j = 0; j < classes; ++j){
if (dets[i].prob[j] > thresh){
if (class < 0) {
//strcat(labelstr, names[j]); // deleted
sprintf((labelstr+p), "%s %.1f%%", names[j], dets[i].prob[j]*100.0); // added
class = j;
} else {
//strcat(labelstr, ", "); // deleted
//strcat(labelstr, names[j]); // deleted
sprintf((labelstr+p), "%s %.1f%%, ", names[j], dets[i].prob[j]*100.0); // added
}
p = strlen(labelstr); // added
printf("%s: %.0f%%\n", names[j], dets[i].prob[j]*100);
}
}
...
include/darknet.h
As a result of the two functions added above to the src/image.c file, the following two corresponding prototypes were added for them between the prototypes for load_image_color() and make_image().
image load_image_color(char *filename, int w, int h); image load_image_from_data(unsigned char *data, int w, int h); void copy_image_to_data(unsigned char *data, image *im, int w, int h); image make_image(int w, int h, int c);
Makefile
For the Makefile, I set GPU to 1 to tell it to use the GPU and I also added -lrt to link in the standard shm_open() function which I call in the above comm_parse_and_process_shm() function in examples/detector.c.
GPU=1
CUDNN=0
OPENCV=0
OPENMP=0
DEBUG=0
ARCH= -gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=[sm_50,compute_50] \
-gencode arch=compute_52,code=[sm_52,compute_52]
# -gencode arch=compute_20,code=[sm_20,sm_21] \ This one is deprecated?
# This is what I use, uncomment if you know your arch and want to specify
# ARCH= -gencode arch=compute_52,code=compute_52
VPATH=./src/:./examples
SLIB=libdarknet.so
ALIB=libdarknet.a
EXEC=darknet
OBJDIR=./obj/
CC=gcc
CPP=g++
NVCC=nvcc
AR=ar
ARFLAGS=rcs
OPTS=-Ofast
LDFLAGS= -lm -pthread -lrt
