* [PATCH 00/19] another round of httpd improvements
@ 2022-12-15 20:52 Eric Wong
2022-12-15 20:52 ` [PATCH 01/19] mwrap_httpd: show current bytes consistently Eric Wong
` (18 more replies)
0 siblings, 19 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
Eric Wong (19):
mwrap_httpd: show current bytes consistently
introduce AUTO_FREE macro to simplify cleanup
httpd: rework httpd to use auto-free for memstream
httpd: avoid newline if not using bt: >= 1
mwrap_httpd: flesh out /$PID/ and /$PID/trim endpoints
mwrap_httpd: add info about src_file and src_loc stats
use uatomic_inc where appropriate
httpd: drop unnecessary AND ops from base-64
mymalloc: add notes on the malloc implementation
rproxy: link to mwrap_httpd /$PID/ root without each, too
httpd: shrink `mean_life' field to `double'
httpd: support CSV output
rproxy: enable deflater by default
mwrap_httpd: do not abort on fork if out-of-resources
httpd: pause forking thread on resource limitations
rename mwrap_httpd.h to httpd.h
httpd: describe simple and naive buffering scheme
httpd: drop TODO item for pipelining
avoid -Warray-bounds warning, avoid stack overallocation
MANIFEST | 4 +-
Mwrap.xs | 5 +-
mwrap_httpd.h => httpd.h | 467 ++++++++++++++++++++++-------------
lib/Devel/Mwrap/PSGI.pm | 2 +
lib/Devel/Mwrap/Rproxy.pm | 7 +-
mwrap_core.h | 53 +++-
mymalloc.h | 15 +-
script/mwrap-rproxy | 12 +-
t/{mwrap-httpd.t => httpd.t} | 29 +++
9 files changed, 398 insertions(+), 196 deletions(-)
rename mwrap_httpd.h => httpd.h (71%)
rename t/{mwrap-httpd.t => httpd.t} (79%)
^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 01/19] mwrap_httpd: show current bytes consistently
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 02/19] introduce AUTO_FREE macro to simplify cleanup Eric Wong
` (17 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
We actually want to read total_bytes_dec before total_bytes_inc
to ensure a greater likelyhood of the resulting total being
positive (or not underflowed into SIZE_MAX territory)
---
mwrap_httpd.h | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index bd37fb2..aa4574a 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -487,6 +487,13 @@ static struct h1_src_loc *accumulate(unsigned long min, size_t *hslc, FILE *lp)
return hslv;
}
+static void show_age(FILE *fp)
+{
+ size_t dec = uatomic_read(&total_bytes_dec);
+ size_t inc = uatomic_read(&total_bytes_inc);
+ fprintf(fp, "<p>Current age: %zu (live: %zu)", inc , inc - dec);
+}
+
/* /$PID/at/$LOCATION endpoint */
static enum mw_qev each_at(struct mw_h1 *h1, struct mw_h1req *h1r)
{
@@ -519,9 +526,7 @@ static enum mw_qev each_at(struct mw_h1 *h1, struct mw_h1req *h1r)
write_html(fp, lname.ptr, lname.len);
free(lname.ptr);
- size_t age = uatomic_read(&total_bytes_inc);
- fprintf(fp, "<p>Current age: %zu (live: %zu)",
- age, age - uatomic_read(&total_bytes_dec));
+ show_age(fp);
FPUTS("<table><tr><th>size</th><th>generation</th>"
"<th>address</th></tr>", fp);
@@ -580,10 +585,8 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
fprintf(fp, "<html><head><title>mwrap each >%lu"
"</title></head><body><p>mwrap each >%lu "
"(change `%lu' in URL to adjust filtering)", min, min, min);
- size_t age = uatomic_read(&total_bytes_inc);
- fprintf(fp, "<p>Current age: %zu (live: %zu)",
- age, age - uatomic_read(&total_bytes_dec));
+ show_age(fp);
if (bt_req_depth) /* need borders to distinguish multi-level traces */
FPUTS("<table\nborder=1><tr>", fp);
else /* save screen space if only tracing one line */
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 02/19] introduce AUTO_FREE macro to simplify cleanup
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
2022-12-15 20:52 ` [PATCH 01/19] mwrap_httpd: show current bytes consistently Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 03/19] httpd: rework httpd to use auto-free for memstream Eric Wong
` (16 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
Both gcc and clang have had __attribute__((__cleanup__)) for a
decade or more. This helps us avoid memory leaks and simplifies
our code.
---
Mwrap.xs | 3 +--
mwrap_core.h | 14 ++++++++++----
mwrap_httpd.h | 18 +++++-------------
3 files changed, 16 insertions(+), 19 deletions(-)
diff --git a/Mwrap.xs b/Mwrap.xs
index 6adf975..9ebc082 100644
--- a/Mwrap.xs
+++ b/Mwrap.xs
@@ -25,7 +25,7 @@ static SV *location_string(struct src_loc *l)
sv_catpvf(ret, ":%zu", l->lineno);
}
if (l->bt_len) {
- char **s = bt_syms(l->bt, l->bt_len);
+ AUTO_FREE char **s = bt_syms(l->bt, l->bt_len);
if (s) {
if (l->f)
sv_catpvs(ret, "\n");
@@ -33,7 +33,6 @@ static SV *location_string(struct src_loc *l)
for (uint32_t i = 1; i < l->bt_len; ++i)
sv_catpvf(ret, "\n%s", s[i]);
}
- free(s);
}
return ret;
}
diff --git a/mwrap_core.h b/mwrap_core.h
index ec08ee1..2ef550a 100644
--- a/mwrap_core.h
+++ b/mwrap_core.h
@@ -714,6 +714,14 @@ char **bt_syms(void * const *addrlist, uint32_t size)
return s;
}
+/* supported by modern gcc + clang */
+#define AUTO_FREE __attribute__((__cleanup__(cleanup_free)))
+static void cleanup_free(void *any)
+{
+ void **p = any;
+ free(*p);
+}
+
static void *dump_to_file(struct dump_arg *a)
{
struct cds_lfht_iter iter;
@@ -730,13 +738,11 @@ static void *dump_to_file(struct dump_arg *a)
if (l->total <= a->min) continue;
if (loc_is_addr(l)) {
- char **s = bt_syms(l->bt, 1);
+ AUTO_FREE char **s = bt_syms(l->bt, 1);
- if (s) {
+ if (s)
fprintf(a->fp, "%16zu %12zu %s\n",
l->total, l->allocations, s[0]);
- free(s);
- }
} else {
fprintf(a->fp, "%16zu %12zu %s:%zu\n",
l->total, l->allocations, l->f->fn, l->lineno);
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index aa4574a..1f706ea 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -420,7 +420,7 @@ static off_t write_loc_name(FILE *fp, const struct src_loc *l)
fprintf(fp, ":%zu", l->lineno);
}
if (l->bt_len) {
- char **s = bt_syms(l->bt, l->bt_len);
+ AUTO_FREE char **s = bt_syms(l->bt, l->bt_len);
if (!s) return -1;
if (l->f) fputc('\n', fp);
@@ -436,7 +436,6 @@ static off_t write_loc_name(FILE *fp, const struct src_loc *l)
fputc('\n', fp);
fputs(s[i], fp);
}
- free(s);
}
off_t end = ftello(fp);
if (end < 0) {
@@ -560,14 +559,12 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
struct mw_membuf ln;
FILE *lp = memstream_new(&ln);
if (!lp) return h1_close(h1);
- struct h1_src_loc *hslv = accumulate(min, &hslc, lp);
+ AUTO_FREE struct h1_src_loc *hslv = accumulate(min, &hslc, lp);
if (!hslv)
return h1_close(h1);
- if (err_close(lp)) {
- free(hslv);
+ if (err_close(lp))
return h1_close(h1);
- }
char *n = ln.ptr;
for (size_t i = 0; i < hslc; ++i) {
@@ -575,7 +572,6 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
n += hslv[i].lname_len;
if (hslv[i].lname_len < 0) {
free(ln.ptr);
- free(hslv);
return h1_close(h1);
}
}
@@ -628,7 +624,6 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
write_html(fp, hsl->loc_name, hsl->lname_len);
FPUTS("</a></td></tr>", fp);
}
- free(hslv);
free(ln.ptr);
FPUTS("</table></body></html>", fp);
return h1_200(h1, fp, &mb);
@@ -1041,7 +1036,6 @@ static void *h1d_run(void *x) /* pthread_create cb */
{
struct mw_h1d *h1d = x;
nfds_t i, nfds;
- struct pollfd *pfd;
int rc;
struct mw_h1 *h1, *nxt;
enum mw_qev ev;
@@ -1053,7 +1047,7 @@ static void *h1d_run(void *x) /* pthread_create cb */
cds_list_for_each_entry_safe(h1, nxt, &h1d->conn, nd)
if (poll_add(h1d, h1->fd, h1->events))
h1_close(h1);
- pfd = poll_detach(h1d, &nfds);
+ AUTO_FREE struct pollfd *pfd = poll_detach(h1d, &nfds);
rc = pfd ? poll(pfd, nfds, -1) : -1;
if (rc < 0) {
@@ -1083,10 +1077,8 @@ static void *h1d_run(void *x) /* pthread_create cb */
}
}
}
- free(pfd);
}
- pfd = poll_detach(h1d, &nfds);
- free(pfd);
+ free(poll_detach(h1d, &nfds));
cds_list_for_each_entry_safe(h1, nxt, &h1d->conn, nd)
h1_close(h1);
return NULL;
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 03/19] httpd: rework httpd to use auto-free for memstream
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
2022-12-15 20:52 ` [PATCH 01/19] mwrap_httpd: show current bytes consistently Eric Wong
2022-12-15 20:52 ` [PATCH 02/19] introduce AUTO_FREE macro to simplify cleanup Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 04/19] httpd: avoid newline if not using bt: >= 1 Eric Wong
` (15 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
This helps us avoid memory leaks after OOM.
---
mwrap_httpd.h | 154 +++++++++++++++++++++++++-------------------------
1 file changed, 77 insertions(+), 77 deletions(-)
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index 1f706ea..63dfcfa 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -36,9 +36,10 @@ enum mw_qev {
MW_QEV_WR = POLLOUT
};
-struct mw_membuf { /* for open_memstream */
+struct mw_fbuf {
char *ptr;
size_t len;
+ FILE *fp;
};
struct mw_wbuf { /* for response headers + bodies */
@@ -78,8 +79,7 @@ struct mw_h1d { /* the daemon + listener, a singleton */
unsigned alive;
struct cds_list_head conn; /* <=> mw_h1.nd */
/* use open_memstream + fwrite to implement a growing pollfd array */
- FILE *pfp; /* see poll_add, poll_detach */
- struct mw_membuf pbuf; /* pollfd vector */
+ struct mw_fbuf pb; /* pollfd vector */
pthread_t tid;
size_t pid_len;
char pid_str[10];
@@ -202,60 +202,70 @@ static enum mw_qev h1_send_flush(struct mw_h1 *h1)
return h1->persist ? MW_QEV_RD : h1_close(h1);
}
-static FILE *memstream_new(struct mw_membuf *mb)
+static FILE *fbuf_init(struct mw_fbuf *fb)
{
- FILE *fp = open_memstream(&mb->ptr, &mb->len);
- if (!fp) fprintf(stderr, "open_memstream: %m\n");
- return fp;
+ fb->ptr = NULL;
+ fb->fp = open_memstream(&fb->ptr, &fb->len);
+ if (!fb->fp) fprintf(stderr, "open_memstream: %m\n");
+ return fb->fp;
}
-static FILE *wbuf_new(struct mw_membuf *mb)
+static FILE *wbuf_init(struct mw_fbuf *fb)
{
static const struct mw_wbuf pad;
- FILE *fp = memstream_new(mb);
- if (fp) /* pad space is populated before h1_send_flush */
- fwrite(&pad, 1, sizeof(pad), fp);
- return fp;
+ if (fbuf_init(fb)) /* pad space is populated before h1_send_flush */
+ fwrite(&pad, 1, sizeof(pad), fb->fp);
+ return fb->fp;
}
-static int err_close(FILE *fp)
+static int fbuf_close(struct mw_fbuf *fb)
{
- int e = ferror(fp) | fclose(fp);
+ int e = ferror(fb->fp) | fclose(fb->fp);
+ fb->fp = NULL;
if (e) fprintf(stderr, "ferror|fclose: %m\n");
return e;
}
+/* supported by modern gcc + clang */
+#define AUTO_CLOFREE __attribute__((__cleanup__(cleanup_clofree)))
+static void cleanup_clofree(void *ptr)
+{
+ struct mw_fbuf *fb = ptr;
+ if (fb->fp) fclose(fb->fp);
+ free(fb->ptr);
+}
+
static enum mw_qev h1_res_oneshot(struct mw_h1 *h1, const char *buf, size_t len)
{
- struct mw_membuf mb;
- FILE *fp = wbuf_new(&mb);
+ struct mw_fbuf fb;
- if (!fp)
+ if (!wbuf_init(&fb))
return h1_close(h1);
- fwrite(buf, 1, len, fp);
- if (err_close(fp))
+
+ fwrite(buf, 1, len, fb.fp);
+ if (fbuf_close(&fb))
return h1_close(h1);
- /* fill in the zero padding we added at wbuf_new */
+ /* fill in the zero padding we added at wbuf_init */
mwrap_assert(!h1->wbuf);
- struct mw_wbuf *wbuf = h1->wbuf = (struct mw_wbuf *)mb.ptr;
+ struct mw_wbuf *wbuf = h1->wbuf = (struct mw_wbuf *)fb.ptr;
wbuf->iov_nr = 1;
- wbuf->iov[0].iov_len = mb.len - sizeof(*wbuf);
+ wbuf->iov[0].iov_len = fb.len - sizeof(*wbuf);
wbuf->iov[0].iov_base = wbuf->bytes;
return h1_send_flush(h1);
}
#define FPUTS(STR, fp) fwrite(STR, sizeof(STR) - 1, 1, fp)
-static enum mw_qev h1_200(struct mw_h1 *h1, FILE *fp, struct mw_membuf *mb)
+static enum mw_qev h1_200(struct mw_h1 *h1, struct mw_fbuf *fb)
{
/*
* the HTTP header goes at the END of the body buffer,
* we'll rely on iovecs via sendmsg(2) to reorder and clamp it
*/
- off_t clen = ftello(fp);
+ off_t clen = ftello(fb->fp);
if (clen < 0) {
fprintf(stderr, "ftello: %m\n");
- fclose(fp);
+ fbuf_close(fb);
return h1_close(h1);
}
clen -= sizeof(struct mw_wbuf);
@@ -265,18 +275,18 @@ static enum mw_qev h1_200(struct mw_h1 *h1, FILE *fp, struct mw_membuf *mb)
"Pragma: no-cache\r\n"
"Cache-Control: no-cache, max-age=0, must-revalidate\r\n"
"Content-Type: text/html; charset=UTF-8\r\n"
- "Content-Length: ", fp);
- fprintf(fp, "%zu", (size_t)clen);
- FPUTS("\r\n\r\n", fp);
+ "Content-Length: ", fb->fp);
+ fprintf(fb->fp, "%zu", (size_t)clen);
+ FPUTS("\r\n\r\n", fb->fp);
- if (err_close(fp))
+ if (fbuf_close(fb))
return h1_close(h1);
- /* fill in the zero-padding we added at wbuf_new */
+ /* fill in the zero-padding we added at wbuf_init */
mwrap_assert(!h1->wbuf);
- struct mw_wbuf *wbuf = h1->wbuf = (struct mw_wbuf *)mb->ptr;
+ struct mw_wbuf *wbuf = h1->wbuf = (struct mw_wbuf *)fb->ptr;
wbuf->iov_nr = 2;
- wbuf->iov[0].iov_len = mb->len - ((size_t)clen + sizeof(*wbuf));
+ wbuf->iov[0].iov_len = fb->len - ((size_t)clen + sizeof(*wbuf));
wbuf->iov[0].iov_base = wbuf->bytes + (size_t)clen;
wbuf->iov[1].iov_len = clen;
wbuf->iov[1].iov_base = wbuf->bytes;
@@ -447,9 +457,8 @@ static off_t write_loc_name(FILE *fp, const struct src_loc *l)
static struct h1_src_loc *accumulate(unsigned long min, size_t *hslc, FILE *lp)
{
- struct mw_membuf mb;
- FILE *fp = memstream_new(&mb);
- if (!fp) return NULL;
+ struct mw_fbuf fb;
+ if (!fbuf_init(&fb)) return NULL;
rcu_read_lock();
struct cds_lfht *t = CMM_LOAD_SHARED(totals);
struct cds_lfht_iter iter;
@@ -471,17 +480,17 @@ static struct h1_src_loc *accumulate(unsigned long min, size_t *hslc, FILE *lp)
hsl.max_life = uatomic_read(&l->max_lifespan);
hsl.sl = l;
hsl.lname_len = write_loc_name(lp, l);
- fwrite(&hsl, sizeof(hsl), 1, fp);
+ fwrite(&hsl, sizeof(hsl), 1, fb.fp);
}
rcu_read_unlock();
struct h1_src_loc *hslv;
- if (err_close(fp)) {
+ if (fbuf_close(&fb)) {
hslv = NULL;
} else {
- *hslc = mb.len / sizeof(*hslv);
- mwrap_assert((mb.len % sizeof(*hslv)) == 0);
- hslv = (struct h1_src_loc *)mb.ptr;
+ *hslc = fb.len / sizeof(*hslv);
+ mwrap_assert((fb.len % sizeof(*hslv)) == 0);
+ hslv = (struct h1_src_loc *)fb.ptr;
}
return hslv;
}
@@ -507,23 +516,21 @@ static enum mw_qev each_at(struct mw_h1 *h1, struct mw_h1req *h1r)
if (!l) return h1_404(h1);
- struct mw_membuf lname;
- FILE *lp = memstream_new(&lname);
- if (!lp) return h1_close(h1);
- if (write_loc_name(lp, l) < 0) return h1_close(h1);
- if (err_close(lp))
+ AUTO_CLOFREE struct mw_fbuf lb;
+ if (!fbuf_init(&lb)) return h1_close(h1);
+ if (write_loc_name(lb.fp, l) < 0) return h1_close(h1);
+ if (fbuf_close(&lb))
return h1_close(h1);
- struct mw_membuf mb;
- FILE *fp = wbuf_new(&mb);
+ struct mw_fbuf html;
+ FILE *fp = wbuf_init(&html);
if (!fp) return h1_close(h1);
FPUTS("<html><head><title>", fp);
- write_html(fp, lname.ptr, lname.len);
+ write_html(fp, lb.ptr, lb.len);
FPUTS("</title></head><body><p>live allocations at", fp);
if (bt_req_depth) FPUTS("<br/>", fp);
else fputc('\n', fp);
- write_html(fp, lname.ptr, lname.len);
- free(lname.ptr);
+ write_html(fp, lb.ptr, lb.len);
show_age(fp);
FPUTS("<table><tr><th>size</th><th>generation</th>"
@@ -539,7 +546,7 @@ static enum mw_qev each_at(struct mw_h1 *h1, struct mw_h1req *h1r)
}
rcu_read_unlock();
FPUTS("</table></body></html>", fp);
- return h1_200(h1, fp, &mb);
+ return h1_200(h1, &html);
}
/* /$PID/each/$MIN endpoint */
@@ -556,28 +563,26 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
}
size_t hslc;
- struct mw_membuf ln;
- FILE *lp = memstream_new(&ln);
- if (!lp) return h1_close(h1);
- AUTO_FREE struct h1_src_loc *hslv = accumulate(min, &hslc, lp);
+ AUTO_CLOFREE struct mw_fbuf lb;
+ if (!fbuf_init(&lb)) return h1_close(h1);
+ AUTO_FREE struct h1_src_loc *hslv = accumulate(min, &hslc, lb.fp);
if (!hslv)
return h1_close(h1);
- if (err_close(lp))
+ if (fbuf_close(&lb))
return h1_close(h1);
- char *n = ln.ptr;
+ char *n = lb.ptr;
for (size_t i = 0; i < hslc; ++i) {
hslv[i].loc_name = n;
n += hslv[i].lname_len;
- if (hslv[i].lname_len < 0) {
- free(ln.ptr);
+ if (hslv[i].lname_len < 0)
return h1_close(h1);
- }
}
- struct mw_membuf mb;
- FILE *fp = wbuf_new(&mb);
+ struct mw_fbuf html;
+ FILE *fp = wbuf_init(&html);
+ if (!fp) return h1_close(h1);
fprintf(fp, "<html><head><title>mwrap each >%lu"
"</title></head><body><p>mwrap each >%lu "
"(change `%lu' in URL to adjust filtering)", min, min, min);
@@ -624,9 +629,8 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
write_html(fp, hsl->loc_name, hsl->lname_len);
FPUTS("</a></td></tr>", fp);
}
- free(ln.ptr);
FPUTS("</table></body></html>", fp);
- return h1_200(h1, fp, &mb);
+ return h1_200(h1, &html);
}
static enum mw_qev h1_dispatch(struct mw_h1 *h1, struct mw_h1req *h1r)
@@ -864,13 +868,11 @@ static int poll_add(struct mw_h1d *h1d, int fd, short events)
{
struct pollfd pfd;
- if (!h1d->pfp) {
- h1d->pfp = memstream_new(&h1d->pbuf);
- if (!h1d->pfp) return -1;
- }
+ if (!h1d->pb.fp && !fbuf_init(&h1d->pb))
+ return -1;
pfd.fd = fd;
pfd.events = events;
- fwrite(&pfd, 1, sizeof(pfd), h1d->pfp);
+ fwrite(&pfd, 1, sizeof(pfd), h1d->pb.fp);
return 0; /* success */
}
@@ -879,20 +881,18 @@ static struct pollfd *poll_detach(struct mw_h1d *h1d, nfds_t *nfds)
struct pollfd *pfd = NULL; /* our return value */
/* not sure how to best recover from ENOMEM errors in stdio */
- if (h1d->pfp) {
- if (err_close(h1d->pfp)) {
+ if (h1d->pb.fp) {
+ if (fbuf_close(&h1d->pb)) {
exit(EXIT_FAILURE);
} else {
- mwrap_assert(h1d->pbuf.len % sizeof(*pfd) == 0);
- pfd = (struct pollfd *)h1d->pbuf.ptr;
- *nfds = h1d->pbuf.len / sizeof(*pfd);
+ mwrap_assert(h1d->pb.len % sizeof(*pfd) == 0);
+ pfd = (struct pollfd *)h1d->pb.ptr;
+ *nfds = h1d->pb.len / sizeof(*pfd);
}
}
/* prepare a new poll buffer the next loop */
- h1d->pbuf.len = 0;
- h1d->pbuf.ptr = NULL;
- h1d->pfp = NULL;
+ memset(&h1d->pb, 0, sizeof(h1d->pb));
return pfd;
}
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 04/19] httpd: avoid newline if not using bt: >= 1
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (2 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 03/19] httpd: rework httpd to use auto-free for memstream Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 05/19] mwrap_httpd: flesh out /$PID/ and /$PID/trim endpoints Eric Wong
` (14 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
No need to hog up screen space if we're not doing multi-level
traces.
---
mwrap_httpd.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index 63dfcfa..1ef21e0 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -527,9 +527,9 @@ static enum mw_qev each_at(struct mw_h1 *h1, struct mw_h1req *h1r)
if (!fp) return h1_close(h1);
FPUTS("<html><head><title>", fp);
write_html(fp, lb.ptr, lb.len);
- FPUTS("</title></head><body><p>live allocations at", fp);
+ FPUTS("</title></head><body><p>live allocations at:", fp);
if (bt_req_depth) FPUTS("<br/>", fp);
- else fputc('\n', fp);
+ else fputc(' ', fp);
write_html(fp, lb.ptr, lb.len);
show_age(fp);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 05/19] mwrap_httpd: flesh out /$PID/ and /$PID/trim endpoints
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (3 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 04/19] httpd: avoid newline if not using bt: >= 1 Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 06/19] mwrap_httpd: add info about src_file and src_loc stats Eric Wong
` (13 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
I won't be developing Devel::Mwrap::PSGI further since
mwrap_httpd is less intrusive and faster.
---
lib/Devel/Mwrap/PSGI.pm | 2 ++
mwrap_httpd.h | 31 +++++++++++++++++++++++++++++++
t/mwrap-httpd.t | 11 +++++++++++
3 files changed, 44 insertions(+)
diff --git a/lib/Devel/Mwrap/PSGI.pm b/lib/Devel/Mwrap/PSGI.pm
index 5d143cd..3e8795e 100644
--- a/lib/Devel/Mwrap/PSGI.pm
+++ b/lib/Devel/Mwrap/PSGI.pm
@@ -1,5 +1,7 @@
# Copyright (C) all contributors <mwrap@80x24.org>
# License: GPL-2.0+ <https://www.gnu.org/licenses/gpl-2.0.txt>
+#
+# Note: this is deprecated, use mwrap_httpd.h instead
package Devel::Mwrap::PSGI;
use v5.12; # strict
use warnings;
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index 1ef21e0..f5c3e1b 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -29,6 +29,7 @@
#include "picohttpparser_c.h"
#include <pthread.h>
#include <stdbool.h>
+#define URL "https://80x24.org/mwrap-perl.git/about"
enum mw_qev {
MW_QEV_IGNORE = 0,
@@ -321,6 +322,15 @@ static enum mw_qev h1_do_reset(struct mw_h1 *h1)
return h1_res_oneshot(h1, r200, sizeof(r200) - 1);
}
+static enum mw_qev h1_do_trim(struct mw_h1 *h1)
+{
+ static const char r200[] = "HTTP/1.1 200 OK\r\n"
+ "Content-Type: text/plain\r\n"
+ "Content-Length: 9\r\n\r\n" "trimming\n";
+ malloc_trim(0);
+ return h1_res_oneshot(h1, r200, sizeof(r200) - 1);
+}
+
#define PATH_SKIP(h1r, pfx) path_skip(h1r, pfx, sizeof(pfx) - 1)
static const char *path_skip(struct mw_h1req *h1r, const char *pfx, size_t len)
{
@@ -633,6 +643,23 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
return h1_200(h1, &html);
}
+/* /$PID/ root endpoint */
+static enum mw_qev pid_root(struct mw_h1 *h1, struct mw_h1req *h1r)
+{
+ struct mw_fbuf html;
+ FILE *fp = wbuf_init(&html);
+ if (!fp) return h1_close(h1);
+#define default_min "2000"
+
+ FPUTS("<html><head><title>mwrap demo"
+ "</title></head><body><p><a\n"
+ "href=\"each/" default_min "\">allocations >"
+ default_min " bytes</a><p><a\nhref=\""
+ URL "\">" URL "</a></body></html>", fp);
+ return h1_200(h1, &html);
+#undef default_min
+}
+
static enum mw_qev h1_dispatch(struct mw_h1 *h1, struct mw_h1req *h1r)
{
if (h1r->method_len == 3 && !memcmp(h1r->method, "GET", 3)) {
@@ -646,10 +673,14 @@ static enum mw_qev h1_dispatch(struct mw_h1 *h1, struct mw_h1req *h1r)
return each_gt(h1, h1r, min);
} else if ((PATH_SKIP(h1r, "/at/"))) {
return each_at(h1, h1r);
+ } else if (h1r->path_len == 1 && h1r->path[0] == '/') {
+ return pid_root(h1, h1r);
}
} else if (h1r->method_len == 4 && !memcmp(h1r->method, "POST", 4)) {
if (h1r->path_len == 6 && !memcmp(h1r->path, "/reset", 6))
return h1_do_reset(h1);
+ if (h1r->path_len == 5 && !memcmp(h1r->path, "/trim", 5))
+ return h1_do_trim(h1);
}
return h1_404(h1);
}
diff --git a/t/mwrap-httpd.t b/t/mwrap-httpd.t
index aec1780..f300eae 100644
--- a/t/mwrap-httpd.t
+++ b/t/mwrap-httpd.t
@@ -65,6 +65,17 @@ SKIP: {
"http://0/$pid/each/2000");
is($rc, 0, 'curl /each');
unlink($cout);
+
+ $rc = system(qw(curl -vsSf --unix-socket), $sock, '-o', $cout,
+ "http://0/$pid/");
+ is($rc, 0, 'curl / (PID root)');
+ like(slurp($cout), qr/<html>/, 'root shown');
+
+ $rc = system(qw(curl -vsSf -XPOST --unix-socket), $sock, '-o', $cout,
+ "http://0/$pid/trim");
+ is($rc, 0, 'curl / (PID root)');
+ like(slurp($cout), qr/trimming/, 'trim started');
+ unlink($cout);
};
{
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 06/19] mwrap_httpd: add info about src_file and src_loc stats
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (4 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 05/19] mwrap_httpd: flesh out /$PID/ and /$PID/trim endpoints Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 07/19] use uatomic_inc where appropriate Eric Wong
` (12 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
This can give users some quick info about their application.
We'll also show the MWRAP=bt: parameter to hint users about
the backtrace depth they expect.
---
mwrap_core.h | 10 +++++++---
mwrap_httpd.h | 25 +++++++++++++++----------
2 files changed, 22 insertions(+), 13 deletions(-)
diff --git a/mwrap_core.h b/mwrap_core.h
index 2ef550a..4dc28d4 100644
--- a/mwrap_core.h
+++ b/mwrap_core.h
@@ -66,7 +66,7 @@ typedef void COP;
* allocated (and we don't care about overflow on 32-bit since
* hardly anybody still uses it).
*/
-static size_t total_bytes_inc, total_bytes_dec;
+static size_t total_bytes_inc, total_bytes_dec, nr_file, nr_src_loc;
static uint32_t bt_req_depth;
#if MWRAP_PERL
@@ -297,7 +297,9 @@ again:
l->allocations = 1;
CDS_INIT_LIST_HEAD(&l->allocs);
cur = cds_lfht_add_unique(t, l->loc_hash, loc_eq, l, &l->hnode);
- if (cur != &l->hnode) { /* lost race */
+ if (cur == &l->hnode) {
+ uatomic_inc(&nr_src_loc);
+ } else { /* lost race */
rcu_read_unlock();
real_free(l);
rcu_read_lock();
@@ -394,7 +396,9 @@ again:
if (!f) return NULL;
memcpy(f, &sf.sf, sizeof(*f) + len + 1);
cur = cds_lfht_add_unique(t, f->fn_hash, fn_eq, f, &f->nd);
- if (cur != &f->nd) { /* lost race */
+ if (cur == &f->nd) {
+ uatomic_inc(&nr_file);
+ } else { /* lost race */
rcu_read_unlock();
real_free(f);
rcu_read_lock();
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index f5c3e1b..e5790aa 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -505,11 +505,14 @@ static struct h1_src_loc *accumulate(unsigned long min, size_t *hslc, FILE *lp)
return hslv;
}
-static void show_age(FILE *fp)
+static void show_stats(FILE *fp)
{
size_t dec = uatomic_read(&total_bytes_dec);
size_t inc = uatomic_read(&total_bytes_inc);
- fprintf(fp, "<p>Current age: %zu (live: %zu)", inc , inc - dec);
+ fprintf(fp, "<p>Current age: %zu (live: %zu) "
+ "/ files: %zu / locations: %zu",
+ inc , inc - dec,
+ uatomic_read(&nr_file), uatomic_read(&nr_src_loc));
}
/* /$PID/at/$LOCATION endpoint */
@@ -542,7 +545,7 @@ static enum mw_qev each_at(struct mw_h1 *h1, struct mw_h1req *h1r)
else fputc(' ', fp);
write_html(fp, lb.ptr, lb.len);
- show_age(fp);
+ show_stats(fp);
FPUTS("<table><tr><th>size</th><th>generation</th>"
"<th>address</th></tr>", fp);
@@ -595,9 +598,10 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
if (!fp) return h1_close(h1);
fprintf(fp, "<html><head><title>mwrap each >%lu"
"</title></head><body><p>mwrap each >%lu "
- "(change `%lu' in URL to adjust filtering)", min, min, min);
+ "(change `%lu' in URL to adjust filtering) - MWRAP=bt:%u",
+ min, min, min, (unsigned)bt_req_depth);
- show_age(fp);
+ show_stats(fp);
if (bt_req_depth) /* need borders to distinguish multi-level traces */
FPUTS("<table\nborder=1><tr>", fp);
else /* save screen space if only tracing one line */
@@ -651,11 +655,12 @@ static enum mw_qev pid_root(struct mw_h1 *h1, struct mw_h1req *h1r)
if (!fp) return h1_close(h1);
#define default_min "2000"
- FPUTS("<html><head><title>mwrap demo"
- "</title></head><body><p><a\n"
- "href=\"each/" default_min "\">allocations >"
- default_min " bytes</a><p><a\nhref=\""
- URL "\">" URL "</a></body></html>", fp);
+ FPUTS("<html><head><title>mwrap demo</title></head><body>"
+ "<p>mwrap demo", fp);
+ show_stats(fp);
+ FPUTS("<p><a\nhref=\"each/" default_min "\">allocations >"
+ default_min " bytes</a>"
+ "<p><a\nhref=\"" URL "\">" URL "</a></body></html>", fp);
return h1_200(h1, &html);
#undef default_min
}
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 07/19] use uatomic_inc where appropriate
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (5 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 06/19] mwrap_httpd: add info about src_file and src_loc stats Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 08/19] httpd: drop unnecessary AND ops from base-64 Eric Wong
` (11 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
It's slightly more readable IMHO, and perhaps it generates
better code on x86 (since URCU has x86-specific paths for it).
---
mwrap_core.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mwrap_core.h b/mwrap_core.h
index 4dc28d4..d798b79 100644
--- a/mwrap_core.h
+++ b/mwrap_core.h
@@ -280,7 +280,7 @@ again:
l = src_loc_get(t, k);
if (l) {
uatomic_add(&l->total, k->total);
- uatomic_add(&l->allocations, 1);
+ uatomic_inc(&l->allocations);
} else {
size_t n = bt_bytelen(k) + sizeof(*k);
struct cds_lfht_node *cur;
@@ -462,7 +462,7 @@ void free(void *p)
uatomic_add(&total_bytes_dec, h->size);
uatomic_add(&l->freed_bytes, h->size);
uatomic_set(&h->size, 0);
- uatomic_add(&l->frees, 1);
+ uatomic_inc(&l->frees);
uatomic_add(&l->age_total, age);
CHECK(int, 0, pthread_mutex_lock(l->mtx));
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 08/19] httpd: drop unnecessary AND ops from base-64
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (6 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 07/19] use uatomic_inc where appropriate Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 09/19] mymalloc: add notes on the malloc implementation Eric Wong
` (10 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
We only use unsigned chars, so we can let truncation do its
thing and not overflow.
---
mwrap_httpd.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index e5790aa..606ae20 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -361,7 +361,7 @@ static void write_b64_url(FILE *fp, const uint8_t *in, size_t len)
"abcdefghijklmnopqrstuvwxyz" "0123456789-_";
uint8_t o[4];
while (len > 3) {
- o[0] = b64[(in[0] >> 2) & 0x3f];
+ o[0] = b64[in[0] >> 2];
o[1] = b64[((in[0] << 4) | (in[1] >> 4)) & 0x3f];
o[2] = b64[((in[1] << 2) | (in[2] >> 6)) & 0x3f];
o[3] = b64[in[2] & 0x3f];
@@ -372,7 +372,7 @@ static void write_b64_url(FILE *fp, const uint8_t *in, size_t len)
if (len) {
size_t i = 2;
- o[0] = b64[(in[0] >> 2) & 0x3f];
+ o[0] = b64[in[0] >> 2];
o[1] = b64[((in[0] << 4) | (--len ? (in[1] >> 4) : 0)) & 0x3f];
if (len)
o[i++] = b64[((in[1] << 2) |
@@ -410,11 +410,11 @@ static bool b64_url_decode(const void *ptr, size_t *len)
case 0: u = c << 2; break;
case 1:
*out++ = u | c >> 4;
- u = (c & 0xf) << 4;
+ u = c << 4;
break;
case 2:
*out++ = u | c >> 2;
- u = (c & 0x3) << 6;
+ u = c << 6;
break;
case 3: *out++ = u | c;
}
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 09/19] mymalloc: add notes on the malloc implementation
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (7 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 08/19] httpd: drop unnecessary AND ops from base-64 Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 10/19] rproxy: link to mwrap_httpd /$PID/ root without each, too Eric Wong
` (9 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
---
mymalloc.h | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/mymalloc.h b/mymalloc.h
index 7d94246..2e0517b 100644
--- a/mymalloc.h
+++ b/mymalloc.h
@@ -6,6 +6,19 @@
* adds wait-free free(3) multi-threading support to avoid contention
* with call_rcu.
+ * The wait-free free(3) implementation was proposed for inclusion into
+ glibc in 2018 and may eventually become part of glibc:
+ https://inbox.sourceware.org/libc-alpha/20180731084936.g4yw6wnvt677miti@dcvr/
+
+ * Arenas are thread-local and returned to a global pool upon thread
+ destruction. This works well for processes with stable thread counts,
+ but wastes memory in processes with unstable thread counts.
+
+ * On Linux with O_TMPFILE support, all allocations are backed by
+ a temporary file (in TMPDIR). This avoids OOM errors on
+ memory-constrained systems due to the higher-than-normal memory
+ usage of mwrap itself.
+
* memalign-family support is ignored (and reimplemented in mwrap_core.h).
dlmalloc's attempts to improve memory-efficiency is prone to fragmentation
if memaligned-allocations are repeatedly freed and relalocated while
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 10/19] rproxy: link to mwrap_httpd /$PID/ root without each, too
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (8 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 09/19] mymalloc: add notes on the malloc implementation Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 11/19] httpd: shrink `mean_life' field to `double' Eric Wong
` (8 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
We'll keep each/2000 linked, but the /$PID/ root may provide a
better jumping point.
---
lib/Devel/Mwrap/Rproxy.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/Devel/Mwrap/Rproxy.pm b/lib/Devel/Mwrap/Rproxy.pm
index 811b503..f7f8a82 100644
--- a/lib/Devel/Mwrap/Rproxy.pm
+++ b/lib/Devel/Mwrap/Rproxy.pm
@@ -46,8 +46,9 @@ sub list {
for (@pids) {
substr($_, -5, 5, ''); # chop off .sock
my $cmd = $valid_pid->($_) // next;
+ print $fh qq(<a\nhref="./$_/">$_</a>/);
$_ .= '/each/2000';
- say $fh qq(<a\nhref="./), $_, '">', $_, "</a>\t", $cmd;
+ say $fh qq(<a\nhref="./), $_, qq(">each/2000</a>\t), $cmd;
}
print $fh '</pre></body></html>';
r(200, $str);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 11/19] httpd: shrink `mean_life' field to `double'
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (9 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 10/19] rproxy: link to mwrap_httpd /$PID/ root without each, too Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 12/19] httpd: support CSV output Eric Wong
` (7 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
`long double' is excessive, not necessary, and a waste of RAM
and locality.
---
mwrap_httpd.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index 606ae20..3d2bf99 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -96,7 +96,7 @@ static MWRAP_TSD struct mw_h1req *tsd_h1r;
/* sortable snapshot version of struct src_loc */
struct h1_src_loc {
- long double mean_life;
+ double mean_life;
size_t bytes;
size_t allocations;
size_t frees;
@@ -484,8 +484,8 @@ static struct h1_src_loc *accumulate(unsigned long min, size_t *hslc, FILE *lp)
hsl.frees = uatomic_read(&l->frees);
hsl.live = hsl.allocations - hsl.frees;
hsl.mean_life = hsl.frees ?
- ((long double)uatomic_read(&l->age_total) /
- (long double)hsl.frees) :
+ ((double)uatomic_read(&l->age_total) /
+ (double)hsl.frees) :
HUGE_VAL;
hsl.max_life = uatomic_read(&l->max_lifespan);
hsl.sl = l;
@@ -630,7 +630,7 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
struct h1_src_loc *hsl = &hslv[i];
fprintf(fp, "<tr><td>%zu</td><td>%zu</td><td>%zu</td>"
- "<td>%zu</td><td>%0.3Lf</td><td>%zu</td>",
+ "<td>%zu</td><td>%0.3f</td><td>%zu</td>",
hsl->bytes, hsl->allocations, hsl->frees,
hsl->live, hsl->mean_life, hsl->max_life);
FPUTS("<td><a\nhref=\"../at/", fp);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 12/19] httpd: support CSV output
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (10 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 11/19] httpd: shrink `mean_life' field to `double' Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 13/19] rproxy: enable deflater by default Eric Wong
` (6 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
CSV is well-supported by SQLite (and many other tools) so it can
be useful for offline analysis.
---
mwrap_httpd.h | 171 ++++++++++++++++++++++++++++++++----------------
t/mwrap-httpd.t | 18 +++++
2 files changed, 134 insertions(+), 55 deletions(-)
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index 3d2bf99..f484bdd 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -30,6 +30,8 @@
#include <pthread.h>
#include <stdbool.h>
#define URL "https://80x24.org/mwrap-perl.git/about"
+#define TYPE_HTML "text/html; charset=UTF-8"
+#define TYPE_CSV "text/csv"
enum mw_qev {
MW_QEV_IGNORE = 0,
@@ -128,7 +130,7 @@ static int cmp_location(const void *x, const void *y)
return strcmp(a->loc_name, b->loc_name);
}
-/* fields for /each/$MIN/ endpoint */
+/* fields for /each/$MIN{,.csv} endpoints */
struct h1_tbl {
const char *fname;
size_t flen;
@@ -257,7 +259,7 @@ static enum mw_qev h1_res_oneshot(struct mw_h1 *h1, const char *buf, size_t len)
}
#define FPUTS(STR, fp) fwrite(STR, sizeof(STR) - 1, 1, fp)
-static enum mw_qev h1_200(struct mw_h1 *h1, struct mw_fbuf *fb)
+static enum mw_qev h1_200(struct mw_h1 *h1, struct mw_fbuf *fb, const char *ct)
{
/*
* the HTTP header goes at the END of the body buffer,
@@ -275,10 +277,8 @@ static enum mw_qev h1_200(struct mw_h1 *h1, struct mw_fbuf *fb)
"Expires: Fri, 01 Jan 1980 00:00:00 GMT\r\n"
"Pragma: no-cache\r\n"
"Cache-Control: no-cache, max-age=0, must-revalidate\r\n"
- "Content-Type: text/html; charset=UTF-8\r\n"
- "Content-Length: ", fb->fp);
- fprintf(fb->fp, "%zu", (size_t)clen);
- FPUTS("\r\n\r\n", fb->fp);
+ "Content-Type: ", fb->fp);
+ fprintf(fb->fp, "%s\r\nContent-Length: %zu\r\n\r\n", ct, (size_t)clen);
if (fbuf_close(fb))
return h1_close(h1);
@@ -354,6 +354,25 @@ static void write_html(FILE *fp, const char *s, size_t len)
}
}
+/*
+ * quotes multi-line backtraces for CSV (and `\' and `"' in case
+ * we encounter nasty file names).
+ */
+static void write_q_csv(FILE *fp, const char *s, size_t len)
+{
+ fputc('"', fp);
+ for (; len--; ++s) {
+ switch (*s) {
+ case '\n': fputs("\\n", fp); break;
+ case '\\': fputs("\\\\", fp); break;
+ case '"': fputs("\\\"", fp); break;
+ default: fputc(*s, fp);
+ }
+ }
+ fputc('"', fp);
+}
+
+
/* URI-safe base-64 (RFC 4648) */
static void write_b64_url(FILE *fp, const uint8_t *in, size_t len)
{
@@ -559,12 +578,12 @@ static enum mw_qev each_at(struct mw_h1 *h1, struct mw_h1req *h1r)
}
rcu_read_unlock();
FPUTS("</table></body></html>", fp);
- return h1_200(h1, &html);
+ return h1_200(h1, &html, TYPE_HTML);
}
/* /$PID/each/$MIN endpoint */
static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
- unsigned long min)
+ unsigned long min, bool csv)
{
static const char default_sort[] = "bytes";
const char *sort = default_sort;
@@ -593,58 +612,86 @@ static enum mw_qev each_gt(struct mw_h1 *h1, struct mw_h1req *h1r,
return h1_close(h1);
}
- struct mw_fbuf html;
- FILE *fp = wbuf_init(&html);
+ struct mw_fbuf bdy;
+ FILE *fp = wbuf_init(&bdy);
if (!fp) return h1_close(h1);
- fprintf(fp, "<html><head><title>mwrap each >%lu"
- "</title></head><body><p>mwrap each >%lu "
- "(change `%lu' in URL to adjust filtering) - MWRAP=bt:%u",
- min, min, min, (unsigned)bt_req_depth);
- show_stats(fp);
- if (bt_req_depth) /* need borders to distinguish multi-level traces */
- FPUTS("<table\nborder=1><tr>", fp);
- else /* save screen space if only tracing one line */
- FPUTS("<table><tr>", fp);
+ if (!csv) {
+ fprintf(fp, "<html><head><title>mwrap each >%lu"
+ "</title></head><body><p>mwrap each >%lu "
+ "(change `%lu' in URL to adjust filtering) - "
+ "MWRAP=bt:%u", min, min, min, (unsigned)bt_req_depth);
+ show_stats(fp);
+ /* need borders to distinguish multi-level traces */
+ if (bt_req_depth)
+ FPUTS("<table\nborder=1><tr>", fp);
+ else /* save screen space if only tracing one line */
+ FPUTS("<table><tr>", fp);
+ }
int (*cmp)(const void *, const void *) = NULL;
- for (size_t i = 0; i < CAA_ARRAY_SIZE(fields); i++) {
- FPUTS("<th>", fp);
- if (fields[i].flen == sort_len &&
- !memcmp(fields[i].fname, sort, sort_len)) {
- cmp = fields[i].cmp;
- fprintf(fp, "<b>%s</b>", fields[i].fname);
- } else {
- fprintf(fp,
- "<a\nhref=\"./%lu?sort=%s\">%s</a>",
- min, fields[i].fname, fields[i].fname);
+ if (csv) {
+ for (size_t i = 0; i < CAA_ARRAY_SIZE(fields); i++) {
+ const char *fn = fields[i].fname;
+ if (i)
+ fputc(',', fp);
+ fputs(fn, fp);
+ if (fields[i].flen == sort_len &&
+ !memcmp(fn, sort, sort_len))
+ cmp = fields[i].cmp;
+ }
+ fputc('\n', fp);
+ } else {
+ for (size_t i = 0; i < CAA_ARRAY_SIZE(fields); i++) {
+ const char *fn = fields[i].fname;
+ FPUTS("<th>", fp);
+ if (fields[i].flen == sort_len &&
+ !memcmp(fn, sort, sort_len)) {
+ cmp = fields[i].cmp;
+ fprintf(fp, "<b>%s</b>", fields[i].fname);
+ } else {
+ fprintf(fp, "<a\nhref=\"./%lu?sort=%s\">%s</a>",
+ min, fn, fn);
+ }
+ FPUTS("</th>", fp);
}
- FPUTS("</th>", fp);
}
- FPUTS("</tr>", fp);
+ if (!csv)
+ FPUTS("</tr>", fp);
if (cmp)
qsort(hslv, hslc, sizeof(*hslv), cmp);
- else
+ else if (!csv)
FPUTS("<tr><td>sort= not understood</td></tr>", fp);
- for (size_t i = 0; i < hslc; i++) {
- struct h1_src_loc *hsl = &hslv[i];
-
- fprintf(fp, "<tr><td>%zu</td><td>%zu</td><td>%zu</td>"
- "<td>%zu</td><td>%0.3f</td><td>%zu</td>",
- hsl->bytes, hsl->allocations, hsl->frees,
- hsl->live, hsl->mean_life, hsl->max_life);
- FPUTS("<td><a\nhref=\"../at/", fp);
-
- /* yes, we're writing our memory addresses into the URI */
- write_b64_url(fp, (const void *)&hsl->sl->f,
- src_loc_hash_len(hsl->sl));
-
- FPUTS("\">", fp);
- write_html(fp, hsl->loc_name, hsl->lname_len);
- FPUTS("</a></td></tr>", fp);
+ if (csv) {
+ for (size_t i = 0; i < hslc; i++) {
+ struct h1_src_loc *hsl = &hslv[i];
+
+ fprintf(fp, "%zu,%zu,%zu,%zu,%0.3f,%zu,",
+ hsl->bytes, hsl->allocations, hsl->frees,
+ hsl->live, hsl->mean_life, hsl->max_life);
+ write_q_csv(fp, hsl->loc_name, hsl->lname_len);
+ fputc('\n', fp);
+ }
+ } else {
+ for (size_t i = 0; i < hslc; i++) {
+ struct h1_src_loc *hsl = &hslv[i];
+
+ fprintf(fp, "<tr><td>%zu</td><td>%zu</td><td>%zu</td>"
+ "<td>%zu</td><td>%0.3f</td><td>%zu</td>",
+ hsl->bytes, hsl->allocations, hsl->frees,
+ hsl->live, hsl->mean_life, hsl->max_life);
+ FPUTS("<td><a\nhref=\"../at/", fp);
+
+ write_b64_url(fp, (const void *)&hsl->sl->f,
+ src_loc_hash_len(hsl->sl));
+
+ FPUTS("\">", fp);
+ write_html(fp, hsl->loc_name, hsl->lname_len);
+ FPUTS("</a></td></tr>", fp);
+ }
+ FPUTS("</table></body></html>", fp);
}
- FPUTS("</table></body></html>", fp);
- return h1_200(h1, &html);
+ return h1_200(h1, &bdy, csv ? TYPE_CSV : TYPE_HTML);
}
/* /$PID/ root endpoint */
@@ -661,10 +708,19 @@ static enum mw_qev pid_root(struct mw_h1 *h1, struct mw_h1req *h1r)
FPUTS("<p><a\nhref=\"each/" default_min "\">allocations >"
default_min " bytes</a>"
"<p><a\nhref=\"" URL "\">" URL "</a></body></html>", fp);
- return h1_200(h1, &html);
+ return h1_200(h1, &html, TYPE_HTML);
#undef default_min
}
+/* @e is not NUL-terminated */
+static bool sfx_eq(const char *e, const char *sfx)
+{
+ for (const char *m = sfx; *m; m++, e++)
+ if (*e != *m)
+ return false;
+ return true;
+}
+
static enum mw_qev h1_dispatch(struct mw_h1 *h1, struct mw_h1req *h1r)
{
if (h1r->method_len == 3 && !memcmp(h1r->method, "GET", 3)) {
@@ -672,10 +728,15 @@ static enum mw_qev h1_dispatch(struct mw_h1 *h1, struct mw_h1req *h1r)
if ((c = PATH_SKIP(h1r, "/each/"))) {
errno = 0;
- char *end;
- unsigned long min = strtoul(c, &end, 10);
- if ((*end == ' ' || *end == '?') && !errno)
- return each_gt(h1, h1r, min);
+ char *e;
+ unsigned long min = strtoul(c, &e, 10);
+ if (!errno) {
+ if (*e == ' ' || *e == '?')
+ return each_gt(h1, h1r, min, false);
+ if (sfx_eq(e, ".csv") &&
+ (e[4] == ' ' || e[4] == '?'))
+ return each_gt(h1, h1r, min, true);
+ }
} else if ((PATH_SKIP(h1r, "/at/"))) {
return each_at(h1, h1r);
} else if (h1r->path_len == 1 && h1r->path[0] == '/') {
diff --git a/t/mwrap-httpd.t b/t/mwrap-httpd.t
index f300eae..ca90cf0 100644
--- a/t/mwrap-httpd.t
+++ b/t/mwrap-httpd.t
@@ -134,6 +134,24 @@ SKIP: {
SKIP: {
skip 'no reset w/o curl --unix-socket', 1 if !$curl_unix;
+
+ $rc = system(qw(curl -vsSf --unix-socket), $sock, '-o', $cout,
+ "http://0/$pid/each/100.csv");
+ is($rc, 0, '.csv retrieved') or skip 'CSV failed', 1;
+ my $db = "$mwrap_tmp/t.sqlite3";
+ $rc = system(qw(sqlite3), $db, ".import --csv $cout mwrap_each");
+ if ($rc == -1) {
+ diag 'sqlite3 missing';
+ } else {
+ is($rc, 0, 'sqlite3 import');
+ my $n = `sqlite3 $db 'SELECT COUNT(*) FROM mwrap_each'`;
+ is($?, 0, 'sqlite3 count');
+ my $exp = split(/\n/, slurp($cout));
+ is($n + 1, $exp, 'imported all rows into sqlite');
+ my $n = `sqlite3 $db 'SELECT COUNT(*) FROM mwrap_each'`;
+ # diag `sqlite3 $db .schema`;
+ }
+
$rc = system(qw(curl -vsSf --unix-socket), $sock, '-o', $cout,
'-d', 'x=y', "http://0/$pid/reset");
is($rc, 0, 'curl /reset');
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 13/19] rproxy: enable deflater by default
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (11 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 12/19] httpd: support CSV output Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 14/19] mwrap_httpd: do not abort on fork if out-of-resources Eric Wong
` (5 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
Ideally I want this in the main mwrap_httpd itself, but linking
zlib may conflict with whatever an application uses. This will
be useful for exposing the demo endpoint for public use.
---
script/mwrap-rproxy | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/script/mwrap-rproxy b/script/mwrap-rproxy
index be6dcbe..a5572c5 100644
--- a/script/mwrap-rproxy
+++ b/script/mwrap-rproxy
@@ -8,10 +8,12 @@ use Plack::Runner;
use Getopt::Long qw(:config no_ignore_case no_auto_abbrev pass_through);
my $usage = "$0 --socket-dir=/path/to/socket-dir [PLACKUP_OPTIONS]\n";
my $socket_dir;
-GetOptions('socket-dir=s' => \$socket_dir) or die $usage;
+my $gz = 1;
+GetOptions('socket-dir=s' => \$socket_dir, 'deflate!' => \$gz) or die $usage;
$socket_dir //= ($ENV{MWRAP} // '') =~ m!\bsocket_dir:([^,]+)! ? $1 : undef;
$socket_dir // die $usage;
-my $app = Devel::Mwrap::Rproxy->new($socket_dir);
+my $rproxy = Devel::Mwrap::Rproxy->new($socket_dir);
+my $app = sub { $rproxy->call(@_) };
my $runner = Plack::Runner->new;
$runner->parse_options(@ARGV);
if (($ENV{LISTEN_PID} // 0) == $$) {
@@ -26,4 +28,8 @@ Inherited socket (fd=3) is non-blocking, making it blocking.
$runner->set_options(listen_sock => $s);
}
}
-$runner->run(sub { $app->call(@_) });
+if ($gz) {
+ require Plack::Middleware::Deflater;
+ $app = Plack::Middleware::Deflater->wrap($app);
+}
+$runner->run($app);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 14/19] mwrap_httpd: do not abort on fork if out-of-resources
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (12 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 13/19] rproxy: enable deflater by default Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 15/19] httpd: pause forking thread on resource limitations Eric Wong
` (4 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
Instead, we'll leak memory if the pthread_atfork handlers can't
stop httpd at fork due to resource limitations.
---
mwrap_httpd.h | 23 ++++++++++++++++-------
1 file changed, 16 insertions(+), 7 deletions(-)
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index f484bdd..fc096c0 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -1094,6 +1094,12 @@ static int h1d_init(struct mw_h1d *h1d, const char *menv)
memcpy(h1d->pid_str, p, h1d->pid_len);
if (unlink(sa.un.sun_path) < 0 && errno != ENOENT)
return fprintf(stderr, "unlink(%s): %m\n", sa.un.sun_path);
+ /*
+ * lfd may be >=0 if h1d_stop_join failed in parent and we're now
+ * running in a forked child
+ */
+ if (h1d->lfd >= 0)
+ (void)close(h1d->lfd);
h1d->lfd = socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0);
if (h1d->lfd < 0)
return fprintf(stderr, "socket: %m\n");
@@ -1204,25 +1210,28 @@ static void h1d_stop_join(struct mw_h1d *h1d)
socklen_t len = (socklen_t)sizeof(sa);
int e, sfd;
void *ret;
+#define ERR ": %m (can't stop mwrap-httpd before fork)\n"
mwrap_assert(uatomic_read(&h1d->alive) == 0);
if (getsockname(h1d->lfd, &sa.any, &len) < 0) {
- fprintf(stderr, "getsockname: %m\n");
- abort(); /* TODO: graceful fallback (ENOBUFS) */
+ fprintf(stderr, "getsockname"ERR);
+ return;
}
sfd = socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0);
if (sfd < 0) {
- fprintf(stderr, "socket: %m\n");
- abort(); /* TODO: graceful fallback (ENOMEM, EMFILE, ...) */
+ fprintf(stderr, "socket"ERR);
+ return;
}
if (connect(sfd, &sa.any, len) < 0) {
- fprintf(stderr, "connect: %m\n");
- /* TODO: graceful fallback (EAGAIN, ...) */
+ fprintf(stderr, "connect"ERR);
+ close(sfd);
+ return;
}
+#undef ERR
(void)close(sfd);
e = pthread_join(h1d->tid, &ret);
if (e) {
- fprintf(stderr, "pthread_join: %s\n", strerror(e));
+ fprintf(stderr, "BUG? pthread_join: %s\n", strerror(e));
abort();
}
(void)close(h1d->lfd);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 15/19] httpd: pause forking thread on resource limitations
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (13 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 14/19] mwrap_httpd: do not abort on fork if out-of-resources Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 16/19] rename mwrap_httpd.h to httpd.h Eric Wong
` (3 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
We shouldn't abort the program due to resource limitations.
Instead, the least intrusive course of action is probably to
sleep and wait for other threads in the process to do cleanup
work.
---
mwrap_httpd.h | 70 ++++++++++++++++++++++++++++++---------------------
1 file changed, 42 insertions(+), 28 deletions(-)
diff --git a/mwrap_httpd.h b/mwrap_httpd.h
index fc096c0..4864e72 100644
--- a/mwrap_httpd.h
+++ b/mwrap_httpd.h
@@ -79,7 +79,8 @@ struct mw_h1 { /* each HTTP/1.x client (heap) */
struct mw_h1d { /* the daemon + listener, a singleton */
int lfd;
- unsigned alive;
+ uint8_t alive; /* set by parent */
+ uint8_t running; /* cleared by child */
struct cds_list_head conn; /* <=> mw_h1.nd */
/* use open_memstream + fwrite to implement a growing pollfd array */
struct mw_fbuf pb; /* pollfd vector */
@@ -996,8 +997,7 @@ static struct pollfd *poll_detach(struct mw_h1d *h1d, nfds_t *nfds)
static void non_fatal_pause(const char *fail_fn)
{
- fprintf(stderr,
-"%s: %m (non-fatal, pausing mwrap-httpd thread)\n", fail_fn);
+ fprintf(stderr, "%s: %m (non-fatal, pausing mwrap-httpd)\n", fail_fn);
poll(NULL, 0, 1000);
}
@@ -1037,13 +1037,19 @@ static void h1d_event_step(struct mw_h1d *h1d)
case EPERM:
fail_fn = "accept4";
break;
- /* hope other cleanup work gets done: */
+ /*
+ * EINVAL, EBADF, ENOTSOCK, EOPNOTSUPP are all fatal
+ * bugs. The last 3 would be wayward closes in the
+ * application being traced
+ */
default:
- fprintf(stderr, "accept4: %m (fatal)\n");
+ fprintf(stderr,
+ "accept4: %m (fatal in mwrap-httpd)\n");
abort();
}
}
}
+ /* hope other cleanup work gets done by other threads: */
non_fatal_pause(fail_fn);
}
@@ -1094,12 +1100,6 @@ static int h1d_init(struct mw_h1d *h1d, const char *menv)
memcpy(h1d->pid_str, p, h1d->pid_len);
if (unlink(sa.un.sun_path) < 0 && errno != ENOENT)
return fprintf(stderr, "unlink(%s): %m\n", sa.un.sun_path);
- /*
- * lfd may be >=0 if h1d_stop_join failed in parent and we're now
- * running in a forked child
- */
- if (h1d->lfd >= 0)
- (void)close(h1d->lfd);
h1d->lfd = socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0);
if (h1d->lfd < 0)
return fprintf(stderr, "socket: %m\n");
@@ -1112,6 +1112,7 @@ static int h1d_init(struct mw_h1d *h1d, const char *menv)
goto close_fail;
}
h1d->alive = 1; /* runs in parent, before pthread_create */
+ h1d->running = 1;
CDS_INIT_LIST_HEAD(&h1d->conn);
return 0;
close_fail:
@@ -1145,8 +1146,8 @@ static void *h1d_run(void *x) /* pthread_create cb */
locating = 1; /* don't report our own memory use */
for (; uatomic_read(&h1d->alive); ) {
- if (poll_add(h1d, h1d->lfd, POLLIN))
- exit(EXIT_FAILURE);
+ while (poll_add(h1d, h1d->lfd, POLLIN))
+ non_fatal_pause("poll_add(lfd)");
cds_list_for_each_entry_safe(h1, nxt, &h1d->conn, nd)
if (poll_add(h1d, h1->fd, h1->events))
h1_close(h1);
@@ -1155,13 +1156,14 @@ static void *h1d_run(void *x) /* pthread_create cb */
if (rc < 0) {
switch (errno) {
- case EINTR: break;
+ case EINTR: break; /* shouldn't happen, actually */
case ENOMEM: /* may be common */
case EINVAL: /* RLIMIT_NOFILE hit */
non_fatal_pause("poll");
break; /* to forloop where rc<0 */
- default:
- fprintf(stderr, "poll: %m (fatal)\n");
+ default: /* EFAULT is a fatal bug */
+ fprintf(stderr,
+ "poll: %m (fatal in mwrap-httpd)\n");
abort();
}
} else {
@@ -1181,6 +1183,7 @@ static void *h1d_run(void *x) /* pthread_create cb */
}
}
}
+ uatomic_set(&h1d->running, 0);
free(poll_detach(h1d, &nfds));
cds_list_for_each_entry_safe(h1, nxt, &h1d->conn, nd)
h1_close(h1);
@@ -1210,27 +1213,34 @@ static void h1d_stop_join(struct mw_h1d *h1d)
socklen_t len = (socklen_t)sizeof(sa);
int e, sfd;
void *ret;
-#define ERR ": %m (can't stop mwrap-httpd before fork)\n"
+#define ERR ": (stopping mwrap-httpd before fork): "
mwrap_assert(uatomic_read(&h1d->alive) == 0);
- if (getsockname(h1d->lfd, &sa.any, &len) < 0) {
- fprintf(stderr, "getsockname"ERR);
- return;
+ while (getsockname(h1d->lfd, &sa.any, &len) < 0) {
+ non_fatal_pause("getsockname"ERR);
+ if (!uatomic_read(&h1d->running))
+ goto join_thread;
}
- sfd = socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0);
- if (sfd < 0) {
- fprintf(stderr, "socket"ERR);
- return;
+retry_socket:
+ while ((sfd = socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0)) < 0) {
+ non_fatal_pause("socket"ERR);
+ if (!uatomic_read(&h1d->running))
+ goto join_thread;
}
if (connect(sfd, &sa.any, len) < 0) {
- fprintf(stderr, "connect"ERR);
+ int e = errno;
close(sfd);
- return;
+ errno = e;
+ non_fatal_pause("connect"ERR);
+ if (!uatomic_read(&h1d->running))
+ goto join_thread;
+ goto retry_socket;
}
#undef ERR
(void)close(sfd);
+join_thread:
e = pthread_join(h1d->tid, &ret);
- if (e) {
+ if (e) { /* EDEADLK, EINVAL, ESRCH are all fatal bugs */
fprintf(stderr, "BUG? pthread_join: %s\n", strerror(e));
abort();
}
@@ -1248,8 +1258,12 @@ static void h1d_start(void) /* may be called as pthread_atfork child cb */
{
if (mwrap_env && !h1d_init(&g_h1d, mwrap_env) && g_h1d.alive) {
int rc = pthread_create(&g_h1d.tid, NULL, h1d_run, &g_h1d);
- if (rc) /* non-fatal */
+ if (rc) { /* non-fatal */
fprintf(stderr, "pthread_create: %s\n", strerror(rc));
+ (void)close(g_h1d.lfd);
+ g_h1d.lfd = -1;
+ g_h1d.alive = 0;
+ }
}
}
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 16/19] rename mwrap_httpd.h to httpd.h
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (14 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 15/19] httpd: pause forking thread on resource limitations Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 17/19] httpd: describe simple and naive buffering scheme Eric Wong
` (2 subsequent siblings)
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
No need to prefix our project name into every individual source
file, especially since it's an auxilliary component.
---
MANIFEST | 4 ++--
Mwrap.xs | 2 +-
mwrap_httpd.h => httpd.h | 0
lib/Devel/Mwrap/PSGI.pm | 2 +-
lib/Devel/Mwrap/Rproxy.pm | 4 ++--
mwrap_core.h | 2 +-
mymalloc.h | 2 +-
t/{mwrap-httpd.t => httpd.t} | 0
8 files changed, 8 insertions(+), 8 deletions(-)
rename mwrap_httpd.h => httpd.h (100%)
rename t/{mwrap-httpd.t => httpd.t} (100%)
diff --git a/MANIFEST b/MANIFEST
index 39473a5..4e3c487 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -8,12 +8,12 @@ check.h
dlmalloc_c.h
examples/mwrap.psgi
gcc.h
+httpd.h
jhash.h
lib/Devel/Mwrap.pm
lib/Devel/Mwrap/PSGI.pm
lib/Devel/Mwrap/Rproxy.pm
mwrap_core.h
-mwrap_httpd.h
mymalloc.h
picohttpparser.h
picohttpparser_c.h
@@ -21,7 +21,7 @@ ppport.h
script/mwrap-perl
script/mwrap-rproxy
t/httpd-unit.t
-t/mwrap-httpd.t
+t/httpd.t
t/mwrap.t
t/source_location.perl
t/test_common.perl
diff --git a/Mwrap.xs b/Mwrap.xs
index 9ebc082..568ec2b 100644
--- a/Mwrap.xs
+++ b/Mwrap.xs
@@ -12,7 +12,7 @@
*/
typedef struct src_loc * Devel__Mwrap__SrcLoc;
-/* keep this consistent with mwrap_httpd.h write_loc_name */
+/* keep this consistent with httpd.h write_loc_name */
static SV *location_string(struct src_loc *l)
{
SV *ret = newSV(0);
diff --git a/mwrap_httpd.h b/httpd.h
similarity index 100%
rename from mwrap_httpd.h
rename to httpd.h
diff --git a/lib/Devel/Mwrap/PSGI.pm b/lib/Devel/Mwrap/PSGI.pm
index 3e8795e..f2be733 100644
--- a/lib/Devel/Mwrap/PSGI.pm
+++ b/lib/Devel/Mwrap/PSGI.pm
@@ -1,7 +1,7 @@
# Copyright (C) all contributors <mwrap@80x24.org>
# License: GPL-2.0+ <https://www.gnu.org/licenses/gpl-2.0.txt>
#
-# Note: this is deprecated, use mwrap_httpd.h instead
+# Note: this is deprecated, use httpd.h instead
package Devel::Mwrap::PSGI;
use v5.12; # strict
use warnings;
diff --git a/lib/Devel/Mwrap/Rproxy.pm b/lib/Devel/Mwrap/Rproxy.pm
index f7f8a82..76b7d7f 100644
--- a/lib/Devel/Mwrap/Rproxy.pm
+++ b/lib/Devel/Mwrap/Rproxy.pm
@@ -1,7 +1,7 @@
# Copyright (C) mwrap hackers <mwrap-perl@80x24.org>
# License: GPL-2.0+ <https://www.gnu.org/licenses/gpl-2.0.txt>
-# minimal reverse proxy to expose the embedded mwrap_httpd.h UNIX sockets
+# minimal reverse proxy to expose the embedded httpd.h UNIX sockets
# via PSGI (and thus TCP HTTP/1.x). This does not have a hard dependency
# on Mwrap.so.
#
@@ -69,7 +69,7 @@ sub call { # PSGI entry point
my $h = "$method $uri HTTP/1.0\n\n";
$s = send($c, $h, MSG_NOSIGNAL) // return r(500, "send: $!");
$s == length($h) or return r(500, "send $s <".length($h));
- # this only expects mwrap_httpd.h output, so no continuation lines:
+ # this only expects httpd.h output, so no continuation lines:
$h = do { local $/ = "\r\n\r\n"; <$c> } // return r(500, "read: $!");
my ($code, @hdr) = split(/\r\n/, $h);
(undef, $code, undef) = split(/ /, $code);
diff --git a/mwrap_core.h b/mwrap_core.h
index d798b79..1b3d98f 100644
--- a/mwrap_core.h
+++ b/mwrap_core.h
@@ -958,7 +958,7 @@ static struct src_loc *mwrap_get_bin(const char *buf, size_t len)
}
static const char *mwrap_env;
-#include "mwrap_httpd.h"
+#include "httpd.h"
__attribute__((constructor)) static void mwrap_ctor(void)
{
diff --git a/mymalloc.h b/mymalloc.h
index 2e0517b..518cce3 100644
--- a/mymalloc.h
+++ b/mymalloc.h
@@ -126,7 +126,7 @@ ATTR_COLD static void mstate_tsd_dtor(void *p)
CHECK(int, 0, pthread_mutex_unlock(&global_mtx));
}
-/* see mwrap_httpd.h */
+/* see httpd.h */
static void h1d_atfork_prepare(void);
static void h1d_atfork_parent(void);
static void h1d_start(void);
diff --git a/t/mwrap-httpd.t b/t/httpd.t
similarity index 100%
rename from t/mwrap-httpd.t
rename to t/httpd.t
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 17/19] httpd: describe simple and naive buffering scheme
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (15 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 16/19] rename mwrap_httpd.h to httpd.h Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 18/19] httpd: drop TODO item for pipelining Eric Wong
2022-12-15 20:52 ` [PATCH 19/19] avoid -Warray-bounds warning, avoid stack overallocation Eric Wong
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
Better to have one implementation which works for all clients,
since curl may not be as readily available as socat or a naive
Perl socket implementation.
---
httpd.h | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/httpd.h b/httpd.h
index 4864e72..58db400 100644
--- a/httpd.h
+++ b/httpd.h
@@ -8,6 +8,15 @@
*
* stdio (via open_memstream) is used for all vector management,
* thus everything is a `FILE *'
+ *
+ * Buffering is naive: write in full to a memstream to get an accurate
+ * Content-Length, then write out the header and sendmsg it off.
+ * I'm avoiding a streaming + lazy buffering design based on fopencookie(3)
+ * since that adds more complexity and uses icache.
+ * Supporting gzip would be nice, but linking zlib is not an option since
+ * there's a risk of conflicts if the application links against a different
+ * zlib version. posix_spawn+gzip isn't an option, either, since we don't
+ * want to generate intrusive SIGCHLD.
*/
#ifndef _DEFAULT_SOURCE
# define _DEFAULT_SOURCE
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 18/19] httpd: drop TODO item for pipelining
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (16 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 17/19] httpd: describe simple and naive buffering scheme Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
2022-12-15 20:52 ` [PATCH 19/19] avoid -Warray-bounds warning, avoid stack overallocation Eric Wong
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
It's not worth it. Arguably, supporting persistent connections
for a AF_LOCAL-only daemon is overkill.
---
httpd.h | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/httpd.h b/httpd.h
index 58db400..9b4ebe7 100644
--- a/httpd.h
+++ b/httpd.h
@@ -771,11 +771,10 @@ static enum mw_qev h1_drain_input(struct mw_h1 *h1, struct mw_h1req *h1r)
if (h1r) { /* initial */
ssize_t overread = h1r->rbuf_len - h1r->pret;
mwrap_assert(overread >= 0);
- if ((size_t)overread <= h1->in_len) {
+ if ((size_t)overread <= h1->in_len)
h1->in_len -= overread;
- } else { /* TODO: deal with pipelined requests */
+ else /* pipelining not supported */
return h1_400(h1);
- }
} else { /* continue dealing with a trickle */
h1r = h1->h1r;
mwrap_assert(h1r);
^ permalink raw reply related [flat|nested] 20+ messages in thread
* [PATCH 19/19] avoid -Warray-bounds warning, avoid stack overallocation
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
` (17 preceding siblings ...)
2022-12-15 20:52 ` [PATCH 18/19] httpd: drop TODO item for pipelining Eric Wong
@ 2022-12-15 20:52 ` Eric Wong
18 siblings, 0 replies; 20+ messages in thread
From: Eric Wong @ 2022-12-15 20:52 UTC (permalink / raw)
To: mwrap-perl
Compilers don't like seeing `sl.bt[-1]', so we give backtrace(3)
the same address by another means. We'll also save allocating
one pointer off the stack by capping the union padding.
---
mwrap_core.h | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/mwrap_core.h b/mwrap_core.h
index 1b3d98f..3fd67f1 100644
--- a/mwrap_core.h
+++ b/mwrap_core.h
@@ -134,6 +134,7 @@ static void *my_mempcpy(void *dest, const void *src, size_t n)
#define RETURN_ADDRESS(nr) \
__builtin_extract_return_addr(__builtin_return_address(nr))
+
#define SRC_LOC_BT(bt) union stk_bt bt; do { \
uint32_t depth = locating ? 1 : bt_req_depth; \
switch (depth) { \
@@ -142,7 +143,7 @@ static void *my_mempcpy(void *dest, const void *src, size_t n)
default: /* skip 1st level of BT since thats our function */ \
mwrap_assert(bt_req_depth <= MWRAP_BT_MAX); \
++locating; \
- long n = (long)backtrace(&bt.sl.bt[-1], bt_req_depth); \
+ long n = (long)backtrace(bt_dst(&bt), bt_req_depth); \
--locating; \
bt.sl.bt_len = n <= 1 ? 0 : (uint32_t)n - 1; \
if (n > 1) mwrap_assert(bt.sl.bt[0] == RETURN_ADDRESS(0)); \
@@ -209,9 +210,27 @@ union stk_sf {
union stk_bt {
struct src_loc sl;
- char buf_[sizeof(struct src_loc) + sizeof(void *) * MWRAP_BT_MAX];
+ /* we subtract one level from MWRAP_BT_MAX since we discard one
+ * level of backtrace(3) (see below for why) */
+ char buf_[sizeof(struct src_loc) + sizeof(void *) * (MWRAP_BT_MAX-1)];
};
+/*
+ * we discard the 1st-level of the backtrace(3) since it's our *alloc
+ * function (and therefore uninteresting), so we want backtrace(3) to
+ * write to bt->sl.bt[-1] so that bt->sl.bt[0] is the first interesting
+ * thing.
+ */
+#ifdef static_assert
+static_assert(offsetof(struct src_loc, lineno) + sizeof(void *) ==
+ offsetof(struct src_loc, bt),
+ "bt lineno is is bt[-1]");
+#endif
+static void **bt_dst(union stk_bt *bt)
+{
+ return (void **)&bt->sl.lineno;
+}
+
static struct alloc_hdr *ptr2hdr(void *p)
{
return (struct alloc_hdr *)((uintptr_t)p - sizeof(struct alloc_hdr));
^ permalink raw reply related [flat|nested] 20+ messages in thread
end of thread, other threads:[~2022-12-15 20:52 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-15 20:52 [PATCH 00/19] another round of httpd improvements Eric Wong
2022-12-15 20:52 ` [PATCH 01/19] mwrap_httpd: show current bytes consistently Eric Wong
2022-12-15 20:52 ` [PATCH 02/19] introduce AUTO_FREE macro to simplify cleanup Eric Wong
2022-12-15 20:52 ` [PATCH 03/19] httpd: rework httpd to use auto-free for memstream Eric Wong
2022-12-15 20:52 ` [PATCH 04/19] httpd: avoid newline if not using bt: >= 1 Eric Wong
2022-12-15 20:52 ` [PATCH 05/19] mwrap_httpd: flesh out /$PID/ and /$PID/trim endpoints Eric Wong
2022-12-15 20:52 ` [PATCH 06/19] mwrap_httpd: add info about src_file and src_loc stats Eric Wong
2022-12-15 20:52 ` [PATCH 07/19] use uatomic_inc where appropriate Eric Wong
2022-12-15 20:52 ` [PATCH 08/19] httpd: drop unnecessary AND ops from base-64 Eric Wong
2022-12-15 20:52 ` [PATCH 09/19] mymalloc: add notes on the malloc implementation Eric Wong
2022-12-15 20:52 ` [PATCH 10/19] rproxy: link to mwrap_httpd /$PID/ root without each, too Eric Wong
2022-12-15 20:52 ` [PATCH 11/19] httpd: shrink `mean_life' field to `double' Eric Wong
2022-12-15 20:52 ` [PATCH 12/19] httpd: support CSV output Eric Wong
2022-12-15 20:52 ` [PATCH 13/19] rproxy: enable deflater by default Eric Wong
2022-12-15 20:52 ` [PATCH 14/19] mwrap_httpd: do not abort on fork if out-of-resources Eric Wong
2022-12-15 20:52 ` [PATCH 15/19] httpd: pause forking thread on resource limitations Eric Wong
2022-12-15 20:52 ` [PATCH 16/19] rename mwrap_httpd.h to httpd.h Eric Wong
2022-12-15 20:52 ` [PATCH 17/19] httpd: describe simple and naive buffering scheme Eric Wong
2022-12-15 20:52 ` [PATCH 18/19] httpd: drop TODO item for pipelining Eric Wong
2022-12-15 20:52 ` [PATCH 19/19] avoid -Warray-bounds warning, avoid stack overallocation Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/mwrap-perl.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).