Skip to content

Commit

Permalink
Merge pull request #389 from sisong/dev
Browse files Browse the repository at this point in the history 
support "hdiffz -BSD -SD" create diffFile new format: endsley/bsdiff …
  • Loading branch information
@sisong
sisong authored Jul 12, 2024
2 parents 914c6db + f4302c5 commit b715db2
Show file tree
Hide file tree
Showing 13 changed files with 278 additions and 105 deletions.
9 changes: 9 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

full changelog at: https://github.com/sisong/HDiffPatch/commits

## [v4.7.0](https://github.com/sisong/HDiffPatch/tree/v4.7.0) - 2024-07-12
### Added
* cmdline hdiffz support option "-BSD -SD", to create diffFile compatible with another BSDIFF format "ENDSLEY/BSDIFF43", https://github.com/mendsley/bsdiff ; patch support this format from v4.6.7
* cmdline hdiffz support option "-neq"; if opened, hdiffz will refuse to created diffFile when oldData==newData.
### Fixed
* fixed SFX auto extract logic (SFX executable file is hpatchz file + diffFile)
SFX extract with default option `$selfExtractArchive -f ".\" -X ".\"` when diffFile created by directories;
if diffFile created by empty oldPath, then extract with default option `$selfExtractArchive -f "" -X ".\"`.

## [v4.6.9](https://github.com/sisong/HDiffPatch/tree/v4.6.9) - 2023-12-01
### Added
* cmdline hdiffz & hpatchz support option "-info diffFile", print infos of diffFile;
Expand Down
11 changes: 8 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# [HDiffPatch]
[![release](https://img.shields.io/badge/release-v4.6.10-blue.svg)](https://github.com/sisong/HDiffPatch/releases)
[![release](https://img.shields.io/badge/release-v4.7.0-blue.svg)](https://github.com/sisong/HDiffPatch/releases)
[![license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/sisong/HDiffPatch/blob/master/LICENSE)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-blue.svg)](https://github.com/sisong/HDiffPatch/pulls)
[![+issue Welcome](https://img.shields.io/github/issues-raw/sisong/HDiffPatch?color=green&label=%2Bissue%20welcome)](https://github.com/sisong/HDiffPatch/issues)
Expand Down Expand Up @@ -122,6 +122,8 @@ options:
stepSize>=(1024*4), DEFAULT -SD-256k, recommended 64k,2m etc...
-BSD
create diffFile compatible with bsdiff4, unsupport input directory(folder).
support run with -SD (not used stepSize), then create single compressed
diffFile compatible with endsley/bsdiff (https://github.com/mendsley/bsdiff).
-VCD[-compressLevel[-dictSize]]
create diffFile compatible with VCDIFF, unsupport input directory(folder).
DEFAULT no compress, out format same as $open-vcdiff ... or $xdelta3 -S -e -n ...
Expand Down Expand Up @@ -195,6 +197,9 @@ options:
newManifestFile is created from newPath;
-D force run Directory diff between two files; DEFAULT (no -D) run
directory diff need oldPath or newPath is directory.
-neq
open check: if newPath & oldPath's all datas are equal, then return error;
DEFAULT not check equal.
-d Diff only, do't run patch check, DEFAULT run patch check.
-t Test only, run patch check, patch(oldPath,testDiffFile)==newPath ?
-f Force overwrite, ignore write path already exists;
Expand All @@ -215,7 +220,7 @@ uncompress usage: **hpatchz** [options] **"" diffFile outNewPath**
print info: **hpatchz** -info **diffFile**
create SFX: **hpatchz** [-X-exe#selfExecuteFile] **diffFile -X#outSelfExtractArchive**
run SFX: **selfExtractArchive** [options] **oldPath -X outNewPath**
extract SFX: **selfExtractArchive** (same as: selfExtractArchive -f "" -X "./")
extract SFX: **selfExtractArchive** (same as: $selfExtractArchive -f {""|".\"} -X ".\")
```
if oldPath is empty input parameter ""
options:
Expand Down Expand Up @@ -305,7 +310,7 @@ options:
* **create_lite_diff()**
* **hpatch_lite_open()**
* **hpatch_lite_patch()**
#### bsdiff wrapper API:
#### bsdiff4 & endsley/bsdiff wrapper API:
* **create_bsdiff()**
* **create_bsdiff_stream()**
* **bspatch_with_cache()**
Expand Down
11 changes: 8 additions & 3 deletions README_cn.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# [HDiffPatch]
[![release](https://img.shields.io/badge/release-v4.6.10-blue.svg)](https://github.com/sisong/HDiffPatch/releases)
[![release](https://img.shields.io/badge/release-v4.7.0-blue.svg)](https://github.com/sisong/HDiffPatch/releases)
[![license](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/sisong/HDiffPatch/blob/master/LICENSE)
[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-blue.svg)](https://github.com/sisong/HDiffPatch/pulls)
[![+issue Welcome](https://img.shields.io/github/issues-raw/sisong/HDiffPatch?color=green&label=%2Bissue%20welcome)](https://github.com/sisong/HDiffPatch/issues)
Expand Down Expand Up @@ -122,6 +122,8 @@ $ git clone https://github.com/sisong/bzip2.git ../bzip2
压缩步长stepSize>=(1024*4), 默认为256k, 推荐64k,2m等。
-BSD
创建一个和bsdiff4兼容的补丁, 不支持参数为文件夹。
支持和-SD选项一起运行(不使用其stepSize), 创建单压缩流的补丁文件，
兼容endsley/bsdiff格式 (https://github.com/mendsley/bsdiff)。
-VCD[-compressLevel[-dictSize]]
创建一个标准规范VCDIFF格式的补丁, 不支持参数为文件夹。
默认输出补丁不带压缩, 格式和 $open-vcdiff ... 或 $xdelta3 -S -e -n ... 命令输出的补丁格式兼容；
Expand Down Expand Up @@ -191,6 +193,9 @@ $ git clone https://github.com/sisong/bzip2.git ../bzip2
设置newPath的清单文件newManifestFile;
-D 强制执行文件夹间的diff, 即使输入的是2个文件; 从而为文件间的补丁添加校验功能。
默认情况下oldPath或newPath有一个是文件夹时才会执行文件夹间的diff。
-neq
打开检查: 如果newPath和oldPath的数据都相同，则返回错误；
默认不执行该相等检查。
-d 只执行diff, 不要执行patch检查, 默认会执行patch检查.
-t 只执行patch检查, 检查是否 patch(oldPath,testDiffFile)==newPath ?
-f 强制文件写覆盖, 忽略输出的路径是否已经存在;
Expand All @@ -213,7 +218,7 @@ $ git clone https://github.com/sisong/bzip2.git ../bzip2
(将目标平台的hpatchz可执行文件和补丁包文件合并成一个可执行文件, 称作自释放包SFX)
执行一个自释放包： **selfExtractArchive** [options] **oldPath -X outNewPath**
(利用自释放包来打补丁,将包中自带的补丁数据应用到oldPath上, 合成outNewPath)
执行一个自解压包： **selfExtractArchive** (等价于： selfExtractArchive -f "" -X "./")
执行一个自解压包： **selfExtractArchive** (等价于：$selfExtractArchive -f {""|".\"} -X "./")
```
oldPath可以为空, 输入参数为 ""
选项:
Expand Down Expand Up @@ -301,7 +306,7 @@ $ git clone https://github.com/sisong/bzip2.git ../bzip2
* **create_lite_diff()**
* **hpatch_lite_open()**
* **hpatch_lite_patch()**
#### bsdiff 兼容包装 API:
#### bsdiff4 和 endsley/bsdiff 兼容包装 API:
* **create_bsdiff()**
* **create_bsdiff_stream()**
* **bspatch_with_cache()**
Expand Down
132 changes: 113 additions & 19 deletions bsdiff_wrapper/bsdiff_wrapper.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
#endif
#define _check(value,info) { if (!(value)) { throw std::runtime_error(info); } }
static const char* kBsDiffVersionType="BSDIFF40";
static const char* kEsBsDiffVersionType = "ENDSLEY/BSDIFF43";

namespace hdiff_private{
static inline void pushUInt64(std::vector<unsigned char>& buf,hpatch_uint64_t v){
Expand Down Expand Up @@ -103,23 +104,115 @@ namespace hdiff_private{
}
};


struct TInterlaceStream:public hpatch_TStreamInput{
explicit TInterlaceStream(const TCovers& _covers,TCtrlStream& _ctrlStream,
TNewDataSubDiffStream& _subStream,TNewDataDiffStream& _diffStream)
:curPos(0),curi(0),covers(_covers),
curCtrlPos(0),curSubPos(0),curDiffPos(0),curCtrlLen(0),curSubLen(0),curDiffLen(0),
ctrlStream(_ctrlStream),subStream(_subStream),diffStream(_diffStream){
streamImport=this;
read=_read;
streamSize=ctrlStream.streamSize+subStream.streamSize+diffStream.streamSize;
}
private:
hpatch_StreamPos_t curPos;
size_t curi;
const TCovers& covers;
hpatch_StreamPos_t curCtrlPos;
hpatch_StreamPos_t curSubPos;
hpatch_StreamPos_t curDiffPos;
size_t curCtrlLen;
size_t curSubLen;
size_t curDiffLen;
TCtrlStream& ctrlStream;
TNewDataSubDiffStream& subStream;
TNewDataDiffStream& diffStream;
void _update(){
if (curi+1<covers.coverCount()){
TCover c,cnext;
covers.covers(curi++,&c);
covers.covers(curi,&cnext);
curCtrlLen=3*8;
curDiffLen=cnext.newPos-(c.newPos+c.length);
curSubLen=c.length;
}
}
hpatch_BOOL readTo(hpatch_StreamPos_t readFromPos,unsigned char* out_data,unsigned char* out_data_end){
if (readFromPos!=curPos)
return hpatch_FALSE;
size_t readLen=out_data_end-out_data;
curPos+=readLen;
while (readLen>0){
if (curCtrlLen+curSubLen+curDiffLen==0){
_update();
_check(curCtrlLen+curSubLen+curDiffLen>0,"TInterlaceStream _update");
}

hpatch_StreamPos_t* dataPos;
size_t* dataLen;
hpatch_TStreamInput* dataStream;
if (curCtrlLen>0){
dataPos=&curCtrlPos;
dataLen=&curCtrlLen;
dataStream=&ctrlStream;
}else if (curSubLen>0){
dataPos=&curSubPos;
dataLen=&curSubLen;
dataStream=&subStream;
}else{
assert(curDiffLen>0);
dataPos=&curDiffPos;
dataLen=&curDiffLen;
dataStream=&diffStream;
}

size_t cLen=(*dataLen);
if (cLen>readLen) cLen=readLen;
_check(dataStream->read(dataStream,*dataPos,out_data,out_data+cLen),"TInterlaceStream dataStream->read")
out_data+=cLen;
readLen-=cLen;
(*dataPos)+=cLen;
(*dataLen)-=cLen;
}
return hpatch_TRUE;
}
static hpatch_BOOL _read(const hpatch_TStreamInput* stream,hpatch_StreamPos_t readFromPos,
unsigned char* out_data,unsigned char* out_data_end){
TInterlaceStream* self=(TInterlaceStream*)stream->streamImport;
return self->readTo(readFromPos,out_data,out_data_end);
}
};

static void serialize_bsdiff(const hpatch_TStreamInput* newData,const hpatch_TStreamInput* oldData,
const TCovers& covers,const hpatch_TStreamOutput* out_diff,
const hdiff_TCompress* compressPlugin,bool isZeroSubDiff=false){
const hdiff_TCompress* compressPlugin,bool isEndsleyBsdiff,bool isZeroSubDiff=false){
std::vector<unsigned char> buf;
TDiffStream outDiff(out_diff);
size_t ctrlDataSize_pos;
size_t subDataSize_pos;
{//head
buf.clear();
pushCStr(buf,kBsDiffVersionType);
pushCStr(buf,isEndsleyBsdiff?kEsBsDiffVersionType:kBsDiffVersionType);
ctrlDataSize_pos=buf.size();
pushUInt64(buf,0); //ctrlDataSize
subDataSize_pos=buf.size();
pushUInt64(buf,0); //subDataSize
if (!isEndsleyBsdiff){
pushUInt64(buf,0); //ctrlDataSize
pushUInt64(buf,0); //subDataSize
}
pushUInt64(buf,newData->streamSize);
outDiff.pushBack(buf.data(),buf.size());
}

if (isEndsleyBsdiff){
// endsley/bsdiff
TCtrlStream ctrlStream(covers);
TNewDataSubDiffStream subStream(newData,oldData,covers,true,isZeroSubDiff);
TNewDataDiffStream diffStream(covers,newData);

TInterlaceStream interlaceStream(covers,ctrlStream,subStream,diffStream);
outDiff.pushStream(&interlaceStream,compressPlugin,true);
return;
}
//else bsdiff4

{//ctrl data
TCtrlStream ctrlStream(covers);
Expand All @@ -135,6 +228,7 @@ static void serialize_bsdiff(const hpatch_TStreamInput* newData,const hpatch_TSt
TNewDataSubDiffStream subStream(newData,oldData,covers,true,isZeroSubDiff);
hpatch_StreamPos_t subDataSize=outDiff.pushStream(&subStream,compressPlugin,true);

size_t subDataSize_pos=ctrlDataSize_pos+8;
buf.clear();
pushUInt64(buf,subDataSize);//update subDataSize
_check(out_diff->write(out_diff,subDataSize_pos,buf.data(),
Expand Down Expand Up @@ -173,7 +267,7 @@ static void _to_bsdiff_covers(std::vector<_TCover>& covers,_TSize newSize){
void _create_bsdiff(const unsigned char* newData,const unsigned char* cur_newData_end,const unsigned char* newData_end,
const unsigned char* oldData,const unsigned char* cur_oldData_end,const unsigned char* oldData_end,
const hpatch_TStreamOutput* out_diff,const hdiff_TCompress* compressPlugin,
int kMinSingleMatchScore,bool isUseBigCacheMatch,
bool isEndsleyBsdiff,int kMinSingleMatchScore,bool isUseBigCacheMatch,
ICoverLinesListener* coverLinesListener,size_t threadNum){
std::vector<hpatch_TCover_sz> covers;
get_match_covers_by_sstring(newData,cur_newData_end,oldData,cur_oldData_end,covers,
Expand All @@ -188,7 +282,7 @@ void _create_bsdiff(const unsigned char* newData,const unsigned char* cur_newDat
mem_as_hStreamInput(&newStream,newData,newData_end);
mem_as_hStreamInput(&oldStream,oldData,oldData_end);

serialize_bsdiff(&newStream,&oldStream,_covers,out_diff,compressPlugin);
serialize_bsdiff(&newStream,&oldStream,_covers,out_diff,compressPlugin,isEndsleyBsdiff);
}

}//end namespace hdiff_private
Expand All @@ -198,15 +292,15 @@ using namespace hdiff_private;
void create_bsdiff(const unsigned char* newData,const unsigned char* newData_end,
const unsigned char* oldData,const unsigned char* oldData_end,
const hpatch_TStreamOutput* out_diff,const hdiff_TCompress* compressPlugin,
int kMinSingleMatchScore,bool isUseBigCacheMatch,
bool isEndsleyBsdiff,int kMinSingleMatchScore,bool isUseBigCacheMatch,
ICoverLinesListener* coverLinesListener,size_t threadNum){
_create_bsdiff(newData,newData_end,newData_end,oldData,oldData_end,oldData_end,
out_diff,compressPlugin,kMinSingleMatchScore,isUseBigCacheMatch,
out_diff,compressPlugin,isEndsleyBsdiff,kMinSingleMatchScore,isUseBigCacheMatch,
coverLinesListener,threadNum);
}
void create_bsdiff(const hpatch_TStreamInput* newData,const hpatch_TStreamInput* oldData,
const hpatch_TStreamOutput* out_diff,const hdiff_TCompress* compressPlugin,
int kMinSingleMatchScore,bool isUseBigCacheMatch,
bool isEndsleyBsdiff,int kMinSingleMatchScore,bool isUseBigCacheMatch,
ICoverLinesListener* coverLinesListener,size_t threadNum){
TAutoMem oldAndNewData;
loadOldAndNewStream(oldAndNewData,oldData,newData);
Expand All @@ -215,50 +309,50 @@ void create_bsdiff(const hpatch_TStreamInput* newData,const hpatch_TStreamInput*
unsigned char* pNewData=pOldData+old_size;
unsigned char* pNewDataEnd=pNewData+(size_t)newData->streamSize;
_create_bsdiff(pNewData,pNewDataEnd,pNewDataEnd,pOldData,pOldData+old_size,pOldData+old_size,
out_diff,compressPlugin,kMinSingleMatchScore,isUseBigCacheMatch,
out_diff,compressPlugin,isEndsleyBsdiff,kMinSingleMatchScore,isUseBigCacheMatch,
coverLinesListener,threadNum);
}

void create_bsdiff_stream(const hpatch_TStreamInput* newData,const hpatch_TStreamInput* oldData,
const hpatch_TStreamOutput* out_diff,const hdiff_TCompress* compressPlugin,
size_t kMatchBlockSize,const hdiff_TMTSets_s* mtsets){
bool isEndsleyBsdiff,size_t kMatchBlockSize,const hdiff_TMTSets_s* mtsets){
TCoversBuf covers(newData->streamSize,oldData->streamSize);
get_match_covers_by_block(newData,oldData,&covers,kMatchBlockSize,mtsets);
if (covers._isCover32)
_to_bsdiff_covers(covers.m_covers_limit,(hpatch_uint32_t)newData->streamSize);
else
_to_bsdiff_covers(covers.m_covers_larger,newData->streamSize);
covers.update();
serialize_bsdiff(newData,oldData,covers,out_diff,compressPlugin,true);
serialize_bsdiff(newData,oldData,covers,out_diff,compressPlugin,isEndsleyBsdiff,true);
}


void create_bsdiff_block(unsigned char* newData,unsigned char* newData_end,
unsigned char* oldData,unsigned char* oldData_end,
const hpatch_TStreamOutput* out_diff,const hdiff_TCompress* compressPlugin,
int kMinSingleMatchScore,bool isUseBigCacheMatch,
bool isEndsleyBsdiff,int kMinSingleMatchScore,bool isUseBigCacheMatch,
size_t matchBlockSize,size_t threadNum){
if (matchBlockSize==0){
_create_bsdiff(newData,newData_end,newData_end,oldData,oldData_end,oldData_end,
out_diff,compressPlugin,kMinSingleMatchScore,isUseBigCacheMatch,0,threadNum);
out_diff,compressPlugin,isEndsleyBsdiff,kMinSingleMatchScore,isUseBigCacheMatch,0,threadNum);
return;
}
TCoversOptimMB<TMatchBlock> coversOp(newData,newData_end,oldData,oldData_end,matchBlockSize,threadNum);
_create_bsdiff(newData,coversOp.matchBlock->newData_end_cur,newData_end,
oldData,coversOp.matchBlock->oldData_end_cur,oldData_end,
out_diff,compressPlugin,kMinSingleMatchScore,isUseBigCacheMatch,&coversOp,threadNum);
out_diff,compressPlugin,isEndsleyBsdiff,kMinSingleMatchScore,isUseBigCacheMatch,&coversOp,threadNum);
}
void create_bsdiff_block(const hpatch_TStreamInput* newData,const hpatch_TStreamInput* oldData,
const hpatch_TStreamOutput* out_diff,const hdiff_TCompress* compressPlugin,
int kMinSingleMatchScore,bool isUseBigCacheMatch,
bool isEndsleyBsdiff,int kMinSingleMatchScore,bool isUseBigCacheMatch,
size_t matchBlockSize,size_t threadNum){
TAutoMem oldAndNewData;
loadOldAndNewStream(oldAndNewData,oldData,newData);
size_t old_size=oldData?(size_t)oldData->streamSize:0;
unsigned char* pOldData=oldAndNewData.data();
unsigned char* pNewData=pOldData+old_size;
create_bsdiff_block(pNewData,pNewData+(size_t)newData->streamSize,pOldData,pOldData+old_size,
out_diff,compressPlugin,kMinSingleMatchScore,isUseBigCacheMatch,
out_diff,compressPlugin,isEndsleyBsdiff,kMinSingleMatchScore,isUseBigCacheMatch,
matchBlockSize,threadNum);
}

Expand Down
Loading

0 comments on commit b715db2

Please sign in to comment.