Package: golang.org/x/pkgsite/internal/worker

package worker Import Path golang.org/x/pkgsite/internal/worker (on go.dev) Dependency Relation imports 42 packages, and imported by one package

Involved Source Files fetch.go memory.go metrics.go pages.go d➜ server.go Package worker provides functionality for running a worker service. Its primary operation is to fetch modules from a proxy and write them to the database.

Package-Level Type Names (total 8, in which 3 are exported)

/* sort exporteds by: alphabet | popularity */

type Fetcher (struct) A Fetcher holds state for fetching modules. Fields (total 4, all are exported) Cache *cache.Cache DB *postgres.DB ProxyClient *proxy.Client SourceClient *source.Client Methods (total 4, in which 2 are exported) (*T) FetchAndUpdateLatest(ctx context.Context, modulePath string) (_ *internal.LatestModuleVersions, err error) FetchAndUpdateLatest fetches information about the latest versions from the proxy, and updates the database if the version has changed. It returns the most recent good information, which may be what it just fetched or may be what is already in the DB. It does not update the latest good version; that happens inside InsertModule, because it must be protected by the module-path advisory lock. (*T) FetchAndUpdateState(ctx context.Context, modulePath, requestedVersion, appVersionLabel string) (_ int, resolvedVersion string, err error) FetchAndUpdateState fetches and processes a module version, and then updates the module_version_states table according to the result. It returns an HTTP status code representing the result of the fetch operation, and a non-nil error if this status code is not 200. /* 2 unexporteds ... *//* 2 unexporteds: */ (*T) fetchAndInsertModule(ctx context.Context, modulePath, requestedVersion string, lmv *internal.LatestModuleVersions) *fetchTask fetchAndInsertModule fetches the given module version from the module proxy or (in the case of the standard library) from the Go repo and writes the resulting data to the database. (*T) invalidateCache(ctx context.Context, modulePath string) error invalidateCache deletes the series path for modulePath, as well as any possible URL path of which it is a componentwise prefix. That is, it deletes example.com/mod, example.com/mod@v1.2.3 and example.com/mod/pkg, but not the unrelated example.com/module. We delete the series path, not the module path, because adding a v2 module can affect v1 pages. For example, the first v2 module will add a "higher major version" banner to all v1 pages. While adding a v1 version won't currently affect v2 pages, that could change some day (for instance, if we decide to provide history). So it's better to be safe and delete all paths in the series.

type Server (struct) Server can be installed to serve the go discovery worker. Fields (total 12, none are exported) /* 12 unexporteds ... *//* 12 unexporteds: */ cache *cache.Cache cfg *config.Config db *postgres.DB getExperiments func() []*internal.Experiment indexClient *index.Client proxyClient *proxy.Client queue queue.Queue redisHAClient *redis.Client reportingClient *errorreporting.Client sourceClient *source.Client staticPath template.TrustedSource templates map[string]*template.Template Methods (total 22, in which 1 are exported) (*T) Install(handle func(string, http.Handler)) Install registers server routes using the given handler registration func. /* 21 unexporteds ... *//* 21 unexporteds: */ (*T) clearCache(w http.ResponseWriter, r *http.Request) error (*T) computeProcessingLag(ctx context.Context) (*T) doFetch(w http.ResponseWriter, r *http.Request) (string, int) doFetch executes a fetch request and returns the msg and status. (*T) doIndexPage(w http.ResponseWriter, r *http.Request) (err error) doIndexPage writes the status page. On error it returns the error and a short string to be written back to the client. (*T) doPopulateStdLib(ctx context.Context, suffix string) (string, error) (*T) doVersionsPage(w http.ResponseWriter, r *http.Request) (err error) (*T) errorHandler(f func(w http.ResponseWriter, r *http.Request) error) http.HandlerFunc errorHandler converts a function that returns an error into an http.HandlerFunc. (*T) handleClean(w http.ResponseWriter, r *http.Request) (err error) handleClean handles a request to clean module versions. If the request has a 'limit' query parameter, then up to that many module versions are selected from the DB among those eligible for cleaning, and they are cleaned. If the request has a 'module' query parameter, all versions of that module path are cleaned. It is an error if neither or both query parameters are provided. (*T) handleDelete(w http.ResponseWriter, r *http.Request) error handleDelete deletes the specified module version. (*T) handleEnqueue(w http.ResponseWriter, r *http.Request) (err error) handleEnqueue queries the module_version_states table for the next batch of module versions to process, and enqueues them for processing. Note that this may cause duplicate processing. (*T) handleFetch(w http.ResponseWriter, r *http.Request) handleFetch executes a fetch request and returns a http.StatusOK if the status is not http.StatusInternalServerError, so that the task queue does not retry fetching module versions that have a terminal error. (*T) handleFetchStdMaster(w http.ResponseWriter, r *http.Request) error (*T) handleHTMLPage(f func(w http.ResponseWriter, r *http.Request) error) http.HandlerFunc handleHTMLPage returns an HTML page using a template from s.templates. (*T) handleHealthCheck(w http.ResponseWriter, r *http.Request) (*T) handlePollIndex(w http.ResponseWriter, r *http.Request) (err error) (*T) handlePopulateStdLib(w http.ResponseWriter, r *http.Request) error (*T) handleRepopulateSearchDocuments(w http.ResponseWriter, r *http.Request) error handleRepopulateSearchDocuments repopulates every row in the search_documents table that was last updated before the given time. (*T) handleReprocess(w http.ResponseWriter, r *http.Request) error (*T) handleUpdateImportedByCount(w http.ResponseWriter, r *http.Request) error handleUpdateImportedByCount updates imported_by_count for all packages. (*T) reportError(ctx context.Context, err error, w http.ResponseWriter, r *http.Request) reportError sends the error to the GCP Error Reporting service. TODO(jba): factor out from here and frontend/server.go. (*T) serveError(w http.ResponseWriter, r *http.Request, err error) As Outputs Of (at least one exported) func NewServer(cfg *config.Config, scfg ServerConfig) (_ *Server, err error)

type ServerConfig (struct) ServerConfig contains everything needed by a Server. Fields (total 10, all are exported) DB *postgres.DB GetExperiments func() []*internal.Experiment IndexClient *index.Client ProxyClient *proxy.Client Queue queue.Queue RedisCacheClient *redis.Client RedisHAClient *redis.Client ReportingClient *errorreporting.Client SourceClient *source.Client StaticPath template.TrustedSource As Inputs Of (at least one exported) func NewServer(cfg *config.Config, scfg ServerConfig) (_ *Server, err error)

/* 5 unexporteds ... *//* 5 unexporteds: */

type annotation (struct) Fields (total 2, neither is exported) /* 2 unexporteds ... *//* 2 unexporteds: */ error error msg string Methods (only one, which is exported) ( T) Error() builtin.string Implements (at least one exported) T : error

type fetchTask (struct) fetchTask represents the result of a fetch task that was processed. Fields (total 14, in which 13 are exported) FetchResult fetch.FetchResult FetchResult.Defer func() // caller must defer this on all code paths FetchResult.Error error FetchResult.GoModPath string FetchResult.HasGoMod bool HasGoMod says whether the zip contain a go.mod file. If Module (below) is non-nil, then Module.HasGoMod will be the same value. But HasGoMod will be populated even if Module is nil because there were problems with it, as long as we can download and read the zip. FetchResult.MainVersion string FetchResult.MasterVersion string FetchResult.Module *internal.Module FetchResult.ModulePath string FetchResult.PackageVersionStates []*internal.PackageVersionState FetchResult.RequestedVersion string FetchResult.ResolvedVersion string FetchResult.Status int /* one unexported ... *//* one unexported: */ timings map[string]time.Duration As Outputs Of (at least one unexported) /* at least one unexported ... *//* at least one unexported: */ func (*Fetcher).fetchAndInsertModule(ctx context.Context, modulePath, requestedVersion string, lmv *internal.LatestModuleVersions) *fetchTask As Inputs Of (at least 3, none are exported) /* 3+ unexporteds ... *//* 3+ unexporteds: */ func deleteModule(ctx context.Context, db *postgres.DB, ft *fetchTask) (err error) func logTaskResult(ctx context.Context, ft *fetchTask, prefix string) func updateVersionMap(ctx context.Context, db *postgres.DB, ft *fetchTask) (err error)

type processMemStats (struct) processMemStats holds values that describe the current process's memory. All values are in bytes. Fields (total 2, both are exported) RSS uint64 // resident set size (physical memory in use) VSize uint64 // virtual memory size As Outputs Of (at least one unexported) /* at least one unexported ... *//* at least one unexported: */ func getProcessMemStats() (processMemStats, error)

type serverError (struct) Fields (total 2, neither is exported) /* 2 unexporteds ... *//* 2 unexporteds: */ err error // wrapped error status int // HTTP status code Methods (only one, which is exported) (*T) Error() string Implements (at least one exported) *T : error

type systemMemStats (struct) systemMemStats holds values from the /proc/meminfo file, which describes the total system memory. All values are in bytes. Fields (total 6, all are exported) Available uint64 Buffers uint64 Cached uint64 Free uint64 Total uint64 Used uint64 As Outputs Of (at least one unexported) /* at least one unexported ... *//* at least one unexported: */ func getSystemMemStats() (systemMemStats, error)

Package-Level Functions (total 23, in which 1 are exported)

func NewServer(cfg *config.Config, scfg ServerConfig) (_ *Server, err error) NewServer creates a new Server with the given dependencies.

/* 22 unexporteds ... *//* 22 unexporteds: */

func bytesToMi(b uint64) uint64 bytesToMi converts an integral value of bytes into mebibytes.

func deleteModule(ctx context.Context, db *postgres.DB, ft *fetchTask) (err error)

func env(cfg *config.Config) string

func formatTime(t *time.Time) string

func getCgroupMemStats() map[string]uint64 Read memory information for the current cgroup. (A cgroup is the sandbox in which a docker container runs.) All values are in bytes. Returns nil on any error.

func getCgroupMemStatsErr() (map[string]uint64, error)

func getProcessMemStats() (processMemStats, error)

func getSystemMemStats() (systemMemStats, error) getSystemMemStats reads the /proc/meminfo file to get information about the machine.

func init()

func logTaskResult(ctx context.Context, ft *fetchTask, prefix string)

func parseLimitParam(r *http.Request, defaultValue int) int parseLimitParam parses the query parameter "limit" as an integer. If the parameter is missing or there is a parse error, it is logged and the default value is returned.

func parseModulePathAndVersion(requestPath string) (string, string, error) parseModulePathAndVersion returns the module and version specified by p. p is assumed to have either of the following two structures: - <module>/@v/<version> - <module>/@latest (this is symmetric with the proxy url scheme)

func parseTemplate(staticPath, filename template.TrustedSource) (*template.Template, error) Parse the template for the status page.

func percentage(x, y interface{}) int percentage computes the truncated percentage of x/y. It returns 0 if y is 0. x and y can be any int or uint type.

func recordEnqueue(ctx context.Context, status int)

func recordProcessingLag(ctx context.Context, d time.Duration)

func renderPage(ctx context.Context, w http.ResponseWriter, page interface{}, tmpl *template.Template) (err error)

func resolvedVersion(ctx context.Context, modulePath, requestedVersion string, proxyClient *proxy.Client) string

func shouldDisableProxyFetch(m *internal.ModuleVersionState) bool

func toUint64(n interface{}) uint64

func truncate(length int, text *string) *string

func updateVersionMap(ctx context.Context, db *postgres.DB, ft *fetchTask) (err error)

Package-Level Variables (total 8, in which 3 are exported)

var EnqueueResponseCount *view.View EnqueueResponseCount counts worker enqueue responses by response type.

var ProcessingLag *view.View

var ProxyRemoved map[string]bool ProxyRemoved is a set of module@version that have been removed from the proxy, even though they are still in the index.

/* 5 unexporteds ... *//* 5 unexporteds: */

var enqueueStatus *stats.Int64Measure

var keyEnqueueStatus tag.Key keyEnqueueStatus is a census tag used to keep track of the status of the modules being enqueued.

var locNewYork *time.Location

var processingLag *stats.Int64Measure

var startTime time.Time

Package-Level Constants (total 3, none are exported) /* 3 unexporteds ... *//* 3 unexporteds: */

const cleanDays = 7 Consider a module version for cleaning only if it is older than this.

const indexTemplate = "index.tmpl"

const versionsTemplate = "versions.tmpl"