* Video conference libification eng discussion, this Thursday 16:30 UTC
@ 2023-04-04 17:14 Emily Shaffer
2023-04-11 18:39 ` Emily Shaffer
0 siblings, 1 reply; 7+ messages in thread
From: Emily Shaffer @ 2023-04-04 17:14 UTC (permalink / raw)
To: Git List
Hello folks,
Google is hosting a standing engineering discussion about libifying
Git, for contributors interested in participating in or hearing about
the progress of this effort. Expect this discussion to be free-form
and casual - no powerpoints here!
We're hoping to hold this meeting every Thursday at 9:30am Pacific
(16:30 UTC) via Google Meet.
To get an invite to the video conference and the notes document,
please reply to this email. Please also add points to the agenda
(template follows) if you want to raise awareness of them.
I'll reply to this mail tomorrow-ish with the topic we'll discuss
during the meeting, and I'll reply again afterwards with the notes.
- (asynchronous) What's cooking in libification?
- Patches we sent regarding libification
- Patches for review related to the libification effort
- (asynchronous) What happened in the past 1-2 weeks that interested
parties or intermittent contributors need to know?
- (asynchronous) Where are you stuck? What eng discussions do we want
to have? (We'll choose a topic from this list a day ahead of the
meeting.)
- Session topic: <TBD>
- Emily
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Video conference libification eng discussion, this Thursday 16:30 UTC
2023-04-04 17:14 Emily Shaffer
@ 2023-04-11 18:39 ` Emily Shaffer
0 siblings, 0 replies; 7+ messages in thread
From: Emily Shaffer @ 2023-04-11 18:39 UTC (permalink / raw)
To: Git List
The notes from this session follow:
- (asynchronous) What's cooking in libification?
- Patches we sent regarding libification
- How to review coccinelle?
https://lore.kernel.org/git/kl6l7cuycd3n.fsf@chooglen-macbookpro.roam.corp.google.com
- Patches for review related to the libification effort
- (asynchronous) What happened in the past 1-2 weeks that interested
parties or intermittent contributors need to know?
- (asynchronous) Where are you stuck? What eng discussions do we want
to have? (We'll choose a topic from this list a day ahead of the
meeting.)
* Emily: what's the least ugly way to move existing code into
"libified" code? oldname\_lib.[ch] in the top level? lib/oldname.[ch]?
lib/oldname/oldname.[ch]? Something else?
* Emily/Calvin: How do we feel about
https://github.com/rra/c-tap-harness/ ? (3)
* Calvin: Git-compat-util contains many necessary includes, macros,
and inline functions used across all of git, so therefore any library
must include it. There are a couple of dependencies inside of it,
however, that prevent libraries from compiling independently with it.
(lots)
- Session topic: git-compat-util
* Intros
* Emily: TL git-core @ google
* Jonathan: git-core @ google
* Calvin: git-core @ google
* Randall: Platform maintainer for NonStop
* Cem: Interested in contributing to Git but haven't sent
anything yet, hi 🙂 Curious to hear what's going on and hoping to help
* Siddharth: visiting git-core for just Q2 (@ google)
* Linus: 20%ing on git-core (@ google) - interested in libification stuff
* Glen: git-core @ google
* Randall: Mostly concerned about licensing and whether libification
will change the licensing stuff
* Emily: for now Google doesn't see a reason to try to relicense anything
* Calvin: Git-compat-util contains many necessary includes, macros,
and inline functions used across all of git, so therefore any library
must include it. There are a couple of dependencies inside of it,
however, that prevent libraries from compiling independently with it.
* Everything includes git-compat-util.h. So we need
git-compat-util.h to either be not a catchall (like we did with
cache.h), and should we libify git-compat-util dependencies?
* There are some things that are explicitly compatibility
related, like git\_pread(). This is ifdef'd so we always have a pread
implementation. That implementation comes from wrapper.h, and
wrapper.h/wrapper.c includes Lots Of Stuff :') including trace2 and
usage
* die() impl is pretty bad also
* Jonathan: We could say that git-compat-util.h can't depend on
anything else. For example, we can't die(). This might be good,
because anything provided by the platform shouldn't be die()ing
* Calvin: So that means we don't even want to define usage(),
die(), warn(), error() in git-compat-util.h. That kind of makes sense!
But some libraries want to use die() for now
* Emily: yeah, but we need to refactor that out 🙂
* Calvin: Probably that means we should figure out the error
returning design
* Linus: Are there already circular dependencies here?
* Emily: Does that mean we should build all the
everyone-needs-it libraries into one binary, so they can depend on
each other in a complicated way if they need to?
* Calvin: That doesn't really help with trace2 dependency
* Randall: Are we sure how the packaging is gonna happen? Is it
static, dynamic, …?
* Emily: Within Git codebase, not really that worried yet beyond
making sure they CAN compile independently and unit test
* Jonathan: So mostly focusing on "can I compile this explicit
subset of .[ch] files and run unit tests on them"
* Randall: Similar experience with OpenSSL and they dll-ified
their code. That had a lot of platform-specific pain. Please tell me
if you're interested in dlls 🙂
* Emily: As far as the trace2 stuff, I imagine most people
calling Git libraries from their code won't want to trace at all. So
we probably want to libify trace2 stuff \*and\* provide a stubbed
version of the API, since that's such a common use case
* Calvin: Maybe it makes sense to build that tr2-noop lib right
into a large "git-common-lib"
* Calvin: and we can put strbuf, string\_list, hashmap, etc. in
there and we might even be able to leave some things stubbed in
git\_compat\_util.h….
* Calvin: This combined with packaging everything together into
a common library actually sounds really appealing
* Glen: Do we have a good idea of what everything that would be
included into this common library is? How can we annotate what is and
isn't part of it?
* Randall: A lot of platform problems we have had in the past
are coming from git-compat-util. I'm always worried about it, please
be careful with it 🙂
* Calvin: Hoping to avoid touching anything that's got an #ifdef
as a general rule
* Randall: the atexit() code is a place to be careful for sure.
And NonStop is trying to start using pthreads. So, yes, I'm very
worried here.
* Linus: Would it make sense to have some large "standard
library" file with no deps and start moving stuff into it?
* Emily: I'm worried about that turning into a mega-header like
cache.h (or git-compat-util.h)
* Randall: I like cleaner containment than making things extremely broad
^ permalink raw reply [flat|nested] 7+ messages in thread
* Video conference libification eng discussion, this Thursday 16:30 UTC
@ 2023-04-11 18:40 Emily Shaffer
2023-04-13 17:00 ` Emily Shaffer
0 siblings, 1 reply; 7+ messages in thread
From: Emily Shaffer @ 2023-04-11 18:40 UTC (permalink / raw)
To: Git List
Hello folks,
Google is hosting a standing engineering discussion about libifying
Git, for contributors interested in participating in or hearing about
the progress of this effort. Expect this discussion to be free-form
and casual - no powerpoints here!
We're hoping to hold this meeting every Thursday at 9:30am Pacific
(16:30 UTC) via Google Meet.
To get an invite to the video conference and the notes document,
please reply to this email. Please also add points to the agenda
(template follows) if you want to raise awareness of them.
We'll choose a topic to discuss at the beginning of the meeting, and
I'll reply afterwards with the notes.
- (asynchronous) What's cooking in libification?
- Patches we sent regarding libification
- Patches for review related to the libification effort
- (asynchronous) What happened in the past 1-2 weeks that interested
parties or intermittent contributors need to know?
- (asynchronous) Where are you stuck? What eng discussions do we want
to have? (We'll choose a topic from this list a day ahead of the
meeting.)
- Session topic: <TBD>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Video conference libification eng discussion, this Thursday 16:30 UTC
2023-04-11 18:40 Emily Shaffer
@ 2023-04-13 17:00 ` Emily Shaffer
0 siblings, 0 replies; 7+ messages in thread
From: Emily Shaffer @ 2023-04-13 17:00 UTC (permalink / raw)
To: Git List
The notes from today's session follow.
- (asynchronous) What's cooking in libification?
- Patches we sent regarding libification
- Coccinelle conventions
https://lore.kernel.org/git/pull.1495.git.git.1681329955.gitgitgadget@gmail.com
- Patches for review related to the libification effort
- cache.h cleanups
https://lore.kernel.org/git/pull.1509.v2.git.1680571348.gitgitgadget@gmail.com
- (asynchronous) What happened in the past 1-2 weeks that interested
parties or intermittent contributors need to know?
- (asynchronous) Where are you stuck? What eng discussions do we want
to have? (We'll choose a topic from this list at the beginning of the
meeting.)
- Session topic: <TBD>
* Elijah: I have the next set of cache.h cleanups waiting around,
but I'm waiting for the first set to make it in, and wondering whether
I should rebase onto Calvin
* Calvin: you should send yours without waiting, I don't think
there will be a conflict
* Elijah: ok, will try to avoid strbuf and git-compat-util changes
* Glen: Hey take a look at
https://lore.kernel.org/git/pull.1495.git.git.1681329955.gitgitgadget@gmail.com
* Elijah: I also wasn't sure how to review these, the how-to
sounds useful but I don't know any better either?
* Glen: If this series doesn't seem to make your life easier,
then it hasn't achieved its purpose
* Should this come with MyFirstSemanticPatch..?
* General approval 🙂
* Maybe it can come later
* Calvin: We've been working on the idea of a "git-std-lib" internally
* Libification of "just strbuf" and "just hashmap" vs.
libification of "all the shared low-level stuff that depends on each
other"
* Removing the dependency on wrapper.h/usage.h is really tricky
* So instead, a standard library which is includable from git
library callers and also used throughout Git's own codebase
* Stuff like strbuf, string\_list, abspath, strmap, strvec,
wrapper, usage, utf8, etc etc etc all in a single library
* To get the proof of concept, made some cleanups to
git-compat-util, removed dependencies on repository.h, trace2.h, etc
were refactored or replaced with a stub in the unit test makefile
* Have something promising internally
* Does this seem like a good idea..? Hoping to send RFC after
OOO next week sometime
* Elijah: Expecting lots of people to ask what's the purpose of
the standard library, does it imply backwards compatibility, etc etc
* No backwards compatibility, if consumers want to accept
that churn then they can use it with no promises
* Emily: In fact, it's likely to start out pretty rough,
with still prints and dies and so on, so we will intend to change it a
lot afterwards
* Calvin: You might ask, why not do that first? Mostly, to
unblock others as quickly as possible 🙂 If we can make it possible to
build something on top of it and prove it's usable, then that gives us
more momentum to keep polishing it
* Randall: Do we want to mark these apis as experimental?
* Emily: Yes, this and all the apis we're working on for
the foreseeable future
* Elijah: static? Dynamic?
* Calvin: static for now. I don't have a lot of
experience with Makefile
* Where are these makefile changes going?
* Emily: Early stages, we want to make a guarantee about
the ability to compile and unit test, that's likely to go into
t/Makefile
* Glen: Maybe later it makes sense to move things into
lib/ and stand up a lib/Makefile, but we can cross that bridge when we
come to it
* Randall: atexit() is used in Git, be careful with deps
on atexit() in a dynamic library please! It's not very platform
consistent
* Emily: Randall, can you take one of these sessions and
tell us all the stories about atexit() 🙂
* Elijah: hash.h depending on repository.h, stringmap
depending on hash.h, this is gross. But FYI I did work on something to
remove repository.h from hash.h
* Calvin: we ran into that problem, yeah. Very cool to
hear you have some fix, can we see it 🙂
* Elijah: mempool?
* Calvin: not worried about mempool
* Elijah: Was doing some performance testing moving functions into a
new file and determining if it was helpful or not (to see if I can
replace rust stuff)
* Turns out there's a small performance degradation moving
something into a new file like this?
* We might be able to use this to increase performance in some places too
* Calvin: do we know why?
* Elijah: maybe code locality..? It's a function that's being
passed to qsort, so it was being run in a very tight loop, it seems
like that difference in cache locality made a difference
* Jonathan: I've seen this kind of thing as an argument to be
pro-C++ because templates mean you can choose this functionality at
compile time and avoid this penalty.
* Elijah: I spent a lot of time on microoptimizations for the
merge code, it was difficult to parallelize it as it is (in C/++). Why
not 🦀? Was an interesting experiment to look into performance
regressions. Putting the sort into rust actually gave some speedup.
Being able to optimize that sort was really helpful.
* Randall: Other compilers, like Intel C99, the optimizer can do
some things with comparators in the same object as the qsort
invocation to avoid that penalty.
* Randall: Happy to talk about C multithreading too, look over code, etc
* Randall: Have been pressed into service as CVE analysis manager.
Thinking about the libification project and tests as a very good thing
for tracking down CVEs in the base code (if they lurk there). Being
able to test libraries on their own is really helpful from there,
thanks!
On Tue, Apr 11, 2023 at 11:40 AM Emily Shaffer <nasamuffin@google.com> wrote:
>
> Hello folks,
>
> Google is hosting a standing engineering discussion about libifying
> Git, for contributors interested in participating in or hearing about
> the progress of this effort. Expect this discussion to be free-form
> and casual - no powerpoints here!
>
> We're hoping to hold this meeting every Thursday at 9:30am Pacific
> (16:30 UTC) via Google Meet.
>
> To get an invite to the video conference and the notes document,
> please reply to this email. Please also add points to the agenda
> (template follows) if you want to raise awareness of them.
>
> We'll choose a topic to discuss at the beginning of the meeting, and
> I'll reply afterwards with the notes.
>
> - (asynchronous) What's cooking in libification?
> - Patches we sent regarding libification
> - Patches for review related to the libification effort
> - (asynchronous) What happened in the past 1-2 weeks that interested
> parties or intermittent contributors need to know?
> - (asynchronous) Where are you stuck? What eng discussions do we want
> to have? (We'll choose a topic from this list a day ahead of the
> meeting.)
> - Session topic: <TBD>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Video conference libification eng discussion, this Thursday 16:30 UTC
@ 2023-04-25 21:22 Emily Shaffer
0 siblings, 0 replies; 7+ messages in thread
From: Emily Shaffer @ 2023-04-25 21:22 UTC (permalink / raw)
To: Git List
Hello folks,
Google is hosting a standing engineering discussion about libifying
Git, for contributors interested in participating in or hearing about
the progress of this effort. Expect this discussion to be free-form
and casual - no powerpoints here!
We're hoping to hold this meeting every Thursday at 9:30am Pacific
(16:30 UTC) via Google Meet.
To get an invite to the video conference and the notes document,
please reply to this email. Please also add points to the agenda
(template follows) if you want to raise awareness of them.
We'll choose a topic to discuss at the beginning of the meeting, and
I'll reply afterwards with the notes.
- (asynchronous) What's cooking in libification?
- Patches we sent regarding libification
- Patches for review related to the libification effort
- (asynchronous) What happened in the past 1-2 weeks that interested
parties or intermittent contributors need to know?
- (asynchronous) Where are you stuck? What eng discussions do we want
to have? (We'll choose a topic from this list at the beginning of the
meeting.)
- Session topic: <TBD>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Video conference libification eng discussion, this Thursday 16:30 UTC
@ 2023-05-09 21:08 Emily Shaffer
0 siblings, 0 replies; 7+ messages in thread
From: Emily Shaffer @ 2023-05-09 21:08 UTC (permalink / raw)
To: Git List
Hello folks,
Google is hosting a standing engineering discussion about libifying
Git, for contributors interested in participating in or hearing about
the progress of this effort. Expect this discussion to be free-form
and casual - no powerpoints here!
We're hoping to hold this meeting every Thursday at 9:30am Pacific
(16:30 UTC) via Google Meet.
To get an invite to the video conference and the notes document,
please reply to this email. Please also add points to the agenda
(template follows) if you want to raise awareness of them.
We'll choose a topic to discuss at the beginning of the meeting, and
I'll reply afterwards with the notes.
- (asynchronous) What's cooking in libification?
- Patches we sent regarding libification
- Patches for review related to the libification effort
- (asynchronous) What happened in the past 1-2 weeks that interested
parties or intermittent contributors need to know?
- (asynchronous) Where are you stuck? What eng discussions do we want
to have? (We'll choose a topic from this list at the beginning of the
meeting.)
- Session topic: <TBD>
The notes from last week follow - my bad for not getting them out sooner.
- What's cooking in libification?
- Patches we sent regarding libification
* Strbuf cleanups (all except patch 7/7 has been merged):
https://lore.kernel.org/git/CAFySSZAOqX0pESfJ-7bYg9iR2RDB9PL6sC4Kcvr\_zitdQ1kD\_g@mail.gmail.com/T/
- Patches for review related to the libification effort
- (asynchronous) What happened in the past 1-2 weeks that interested
parties or intermittent contributors need to know?
- Where are you stuck? What eng discussions do we want to have?
(We'll choose a topic from this list at the beginning of the meeting.)
* Calvin: cleaning up git-compat-util. Git-compat-util contains the following:
* Includes for standard and external libraries
* Includes for internal headers (wildmatch.h, banned.h)
* Elijah: I've got a patch removing wildmatch.h - is Calvin
already working on it?
* Calvin: Nope, please send, this is out of scope of
what I was looking at so happy to see your patch removing it
* Declarations for functions in usage.c, wrapper.c, and various
other files
* Conditionally defined includes, functions, variables, macros
* Unconditionally defined includes, functions, variables, macros
* Some of the macros are for allocing arrays. Alloc.h also
contains ALLOC\_GROW and ALLOC\_GROW\_BY. Should those macros be moved
into a header file like alloc-ll.h or a common.h? I originally thought
alloc-ll.h made a lot of sense, but then then are macros like
COPY\_ARRAY and MOVE\_ARRAY that wouldn’t fit in there
* Emily: Would these 4 macros make sense in something
like array.h, instead of packed up with alloc? Does it help at all to
separate this way instead?
* Jonathan: Deciding how to slice it is pretty
subjective, better to focus on what users need to include in order to
use the libraries..?
* But then, moving stuff from usage/wrapper into their
own headers might be a good idea.
* Better to focus on "we need to move it for $REASON"
than for organizational purposes?
* Randall: Historically, the purpose of
git-compat-util.h is to allow all the platform-specific stuff to be in
one place. If we're moving alloc stuff away from there, the alloc
stuff is platform-specific, so it might add some complexity, right?
* Calvin: alloc has #ifdef alloc\_h, i'd like to leave
those platform-specific ifdef'd stuff in git-compat-util, but the
unconditional platform-agnostic stuff might make more sense to stand
alone and reduce the volume of git-compat-util.
* Randall: Yes, having git-compat-util contain only
platform-dependent things.
* Elijah: banned.h is maybe an exception, because it's
required everywhere just like git-compat-util is…?
* Calvin: see point below about git-common-header.h
* I want to lessen the scope of git-compat-util to the following
* Includes for standard and external libraries
* Conditionally defined includes, functions, variables, macros
* Unconditionally defined includes that are directly used by
conditional includes can still be left in here
* Which means the other items belong in different files. I propose:
* common.h
* Unconditionally defined includes, functions, variables, macros
* usage.h, wrapper.h, etc.
* Declarations for functions in usage.c, wrapper.c, and
various other files
* git-common-header.h
* #include "git-compat-util.h"
* #include "usage.h"
* #include "common.h"
* #include "wrapper.h"
* #include "banned.h"
* This will be included by all files instead of
git-compat-util.h (or we could and move git-compat-util.h to a
different file and rename this file to git-compat-util.h)
* Elijah: is wrapper.h/usage.h really needed by everyone?
Otherwise LGTM
* Calvin: usage.h doesn't quite exist, proposing to create
it and start to remove it
* Jonathan: usage.h also has die()
* Emily: +1
* Elijah: next series is basically ready to send, no more cache.h
>:) moves all the header decls into relevant files instead, so there's
nothing left.
* …and there was much rejoicing
* Emily: Josh Steadmon will be joining us working on unit testing
into CI etc, looking forward to it
* Randall: Can share how we run Git suite on NonStop if that's
helpful, since any libification stuff needs to work there
* Emily: Good idea to take a look at Calvin's c-tap-harness RFC,
seems like platform compatibility will be a decision point about how
to make sure unit tests are working on "exotic" platforms
* https://lore.kernel.org/git/20230427175007.902278-1-calvinwan@google.com/
* Elijah: For c-tap-harness, if the unit tests are quite small,
then outputting "test OK" for every test becomes a performance
bottleneck. Are we batching those test plan outputs somehow? Some
other performance thing?
* Emily: benefit of having a source fork is that we can fix it
* Randall: c-tap-harness uses autoconf that's beyond what
NonStop can support. Major issue for NonStop on other projects
* Elijah: but if we're bringing it in as a source dep then
we don't use autoconf and we can just build it with make, right?
* Randall: also isn't it in cpp? Will we be able to use it?
* Calvin: I compiled it as C in the RFC so I think we will be ok
* Randall: oh, yeah, it's C89 so it will be OK for us since
we guarantee C99
* Calvin: no fork up on GH or something, but you should be
able to apply those rfc patches to master and try it out
^ permalink raw reply [flat|nested] 7+ messages in thread
* Video conference libification eng discussion, this Thursday 16:30 UTC
@ 2023-05-30 21:26 Emily Shaffer
0 siblings, 0 replies; 7+ messages in thread
From: Emily Shaffer @ 2023-05-30 21:26 UTC (permalink / raw)
To: Git List
Hello folks,
Google is hosting a standing engineering discussion about libifying
Git, for contributors interested in participating in or hearing about
the progress of this effort. Expect this discussion to be free-form
and casual - no powerpoints here!
We typically hold this meeting every Thursday at 10am Pacific (17:00
UTC) via Google Meet; this week it conflicts with a training at Google
so we're moving it to 16:30 UTC.
To get an invite to the video conference and the notes document,
please reply to this email. Please also add points to the agenda
(template follows) if you want to raise awareness of them.
We'll choose a topic to discuss at the beginning of the meeting, and
I'll reply afterwards with the notes.
* (asynchronous) What's cooking in libification?
* Patches we sent regarding libification
* Patches for review related to the libification effort
* (asynchronous) What happened in the past 1-2 weeks that interested
parties or intermittent contributors need to know?
* (asynchronous) Where are you stuck? What eng discussions do we
want to have? (We'll choose a topic from this list at the beginning of
the meeting.)
* Session topic: TBD
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2023-05-30 21:27 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-30 21:26 Video conference libification eng discussion, this Thursday 16:30 UTC Emily Shaffer
-- strict thread matches above, loose matches on Subject: below --
2023-05-09 21:08 Emily Shaffer
2023-04-25 21:22 Emily Shaffer
2023-04-11 18:40 Emily Shaffer
2023-04-13 17:00 ` Emily Shaffer
2023-04-04 17:14 Emily Shaffer
2023-04-11 18:39 ` Emily Shaffer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).