如果从heritrix3.1.0系统的静态逻辑结构入手，往往看不到系统相关对象的交互作用；如果只从系统的对象动态结构 分析，则又看不到系统的逻辑轮廓

所以源码分析需要动静兼顾，使我们更容易理解它的逻辑与交互，本文采用这个分析方法入手

本文要分析的是spring给Heritrix3.1.0系统bean带来了什么样的管理方式，spring容器的配置文件我们已从上文有了初步的了解

先了解spring容器在系统中是怎样加载配置文件以及怎么初始化的，当我们执行采集任务的build操作时

调用CrawlJob对象的void validateConfiguration()

/*** Does the assembled ApplicationContext self-validate? Any failures* are reported as WARNING log events in the job log. * * TODO: make these severe? */public synchronized void validateConfiguration() {instantiateContainer();if(ac==null) {// fatal errors already encountered and reportedreturn; }ac.validate();HashMap<String,Errors> allErrors = ac.getAllErrors();for(String name : allErrors.keySet()) {for(Object err : allErrors.get(name).getAllErrors()) {LOGGER.log(Level.WARNING,err.toString());}}}

首先加载spring配置文件，初始化spring容器；然后是验证容器

/*** Can the configuration yield an assembled ApplicationContext? */public synchronized void instantiateContainer() {checkXML(); if(ac==null) {try {ac = new PathSharingContext(new String[] {"file:"+primaryConfig.getAbsolutePath()},false,null);ac.addApplicationListener(this);ac.refresh();getCrawlController(); // trigger NoSuchBeanDefinitionException if no CCgetJobLogger().log(Level.INFO,"Job instantiated");} catch (BeansException be) {// Calling doTeardown() and therefore ac.close() here sometimes// triggers an IllegalStateException and logs stack trace from// within spring, even if ac.isActive(). So, just null it.ac = null;beansException(be);}}}

上面方法是装载配置文件，添加CrawlJob对象监听器

Heritrix3.1.0的spring容器是经过系统封装的PathSharingContext对象，PathSharingContext类继承自spring的FileSystemXmlApplicationContext类，在它的构造函数里面传入配置文件

public PathSharingContext(String[] configLocations, boolean refresh, ApplicationContext parent) throws BeansException {super(configLocations, refresh, parent);}

当我们执行采集任务的launch操作时，调用CrawlJob对象的void launch()方法

/*** Launch a crawl into 'running' status, assembling if necessary. * * (Note the crawl may have been configured to start in a 'paused'* state.) */public synchronized void launch() {if (isProfile()) {throw new IllegalArgumentException("Can't launch profile" + this);}if(isRunning()) {getJobLogger().log(Level.SEVERE,"Can't relaunch running job");return;} else {CrawlController cc = getCrawlController();if(cc!=null && cc.hasStarted()) {getJobLogger().log(Level.SEVERE,"Can't relaunch previously-launched assembled job");return;}}validateConfiguration();if(!hasValidApplicationContext()) {getJobLogger().log(Level.SEVERE,"Can't launch problem configuration");return;}//final String job = changeState(j, ACTIVE);// this temporary thread ensures all crawl-created threads// land in the AlertThreadGroup, to assist crawl-wide // logging/alertingalertThreadGroup = new AlertThreadGroup(getShortName());alertThreadGroup.addLogger(getJobLogger());Thread launcher = new Thread(alertThreadGroup, getShortName()+" launchthread") {public void run() {CrawlController cc = getCrawlController();startContext();if(cc!=null) {cc.requestCrawlStart();}}};getJobLogger().log(Level.INFO,"Job launched");scanJobLog();launcher.start();// look busy (and give startContext/crawlStart a chance)try {Thread.sleep(1500);} catch (InterruptedException e) {// do nothing
        }}

这里的重要方法是线程对象里面的void startContext()

/*** Start the context, catching and reporting any BeansExceptions.*/protected synchronized void startContext() {try {ac.start(); // job log file covering just this launch
            getJobLogger().removeHandler(currentLaunchJobLogHandler);File f = new File(ac.getCurrentLaunchDir(), "job.log");currentLaunchJobLogHandler = new FileHandler(f.getAbsolutePath(), true);currentLaunchJobLogHandler.setFormatter(new JobLogFormatter());getJobLogger().addHandler(currentLaunchJobLogHandler);} catch (BeansException be) {doTeardown();beansException(be);} catch (Exception e) {LOGGER.log(Level.SEVERE,e.getClass().getSimpleName()+": "+e.getMessage(),e);try {doTeardown();} catch (Exception e2) {e2.printStackTrace(System.err);}        }}

该方法调用PathSharingContext对象的start方法

 @Overridepublic void start() {initLaunchDir();super.start();}

在上面方法里面，会执行spring容器里面所有bean（实现Lifecycle接口）的start方法

Lifecycle接口声明的方法如下，定义了bean组件的生命周期

public interface Lifecycle {/*** Start this component.* Should not throw an exception if the component is already running.* <p>In the case of a container, this will propagate the start signal* to all components that apply.*/void start();/*** Stop this component.* Should not throw an exception if the component isn't started yet.* <p>In the case of a container, this will propagate the stop signal* to all components that apply.*/void stop();/*** Check whether this component is currently running.* <p>In the case of a container, this will return <code>true</code>* only if <i>all</i> components that apply are currently running.* @return whether the component is currently running*/boolean isRunning();}

从这里我们可以知道，Heritrix3.1.0系统是通过spring容器统一管理bean的生命周期（主要是初始化状态）的 

本文通过打印输出了调用了系统哪些bean的start方法

name:scope
name:loggerModule||org.archive.crawler.reporting.CrawlerLoggerModule
name:scope||org.archive.modules.deciderules.DecideRuleSequencename:candidateScoper
name:candidateScoper||org.archive.crawler.prefetch.CandidateScopername:preparer
name:preparer||org.archive.crawler.prefetch.FrontierPreparername:candidateProcessors
name:candidateProcessors||org.archive.modules.CandidateChainname:preselector
name:preselector||org.archive.crawler.prefetch.MyPreselectorname:preconditions
name:bdb||org.archive.bdb.BdbModule
name:serverCache||org.archive.modules.net.BdbServerCache
name:preconditions||org.archive.crawler.prefetch.PreconditionEnforcername:fetchDns
name:fetchDns||org.archive.modules.fetcher.FetchDNSname:fetchHttp
name:cookieStorage||org.archive.modules.fetcher.BdbCookieStorage
name:fetchHttp||org.archive.modules.fetcher.FetchHTTPname:extractorHttp
name:statisticsTracker||org.archive.crawler.reporting.StatisticsTracker
name:extractorHtml||org.archive.modules.extractor.ExtractorHTML
name:extractorCss||org.archive.modules.extractor.ExtractorCSS
name:extractorJs||org.archive.modules.extractor.ExtractorJS
name:extractorSwf||org.archive.modules.extractor.ExtractorSWF
name:fetchProcessors||org.archive.modules.FetchChain
name:warcWriter||org.archive.modules.writer.MyWriterProcessor
name:candidates||org.archive.crawler.postprocessor.CandidatesProcessor
name:disposition||org.archive.crawler.postprocessor.DispositionProcessor
name:dispositionProcessors||org.archive.modules.DispositionChain
name:crawlController||org.archive.crawler.framework.CrawlController
name:uriUniqFilter||org.archive.crawler.util.BdbUriUniqFilter
name:frontier||org.archive.crawler.frontier.BdbFrontiername:actionDirectory
name:actionDirectory||org.archive.crawler.framework.ActionDirectoryname:checkpointService
name:checkpointService||org.archive.crawler.framework.CheckpointService

--------------------------------------------------------------------------- 

本系列Heritrix 3.1.0 源码解析系本人原创 

转载请注明出处 博客园 刺猬的温驯 

本文链接 http://www.cnblogs.com/chenying99/archive/2013/04/17/3025410.html